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Foreword 



Dear reader, my task in this foreword is to shackle your attention to the challenge of getting 
through Jeffrey Cooper's monograph that follows. 

Your attention is deserved because the subject — what we label with deceptive simplicity 
"intelligence analysis" — is so important and so interesting. The scope of this monograph, like 
that of the analytic profession, is broad and deep, from support to military operations to 
divining the inherently unknowable future of mysterious phenomena, like the political 
prospects of important countries. Jeff Cooper's study, as befits the work of one who has long 
been an acute observer of the Intelligence Community and its work, is packed with critiques, 
observations, and judgments. It would be even more satisfying if the study could be further 
illuminated by clinical case studies of failures and successes. In principle, this lack could be 
remedied if the hurdle of classification could be cleared. In practice, it cannot currently be 
fixed because an adequate body of clinical, diagnostic case studies of both successes and 
failures and lessons learned, particularly from the most relevant, post-Cold War intelligence 
experience, simply does not now exist. Not surprisingly, Mr. Cooper, along with many other 
critics and reformers, such as the Silberman-Robb Commission (of which he was a staff 
member), recommends the institutionalization of a lessons-learned process in our national 
intelligence establishment. This is but one of a rich menu of admonitions to be found in this 
study. 

Mr. Cooper has provided a good, thematic summary of the main points of his monograph. I 
shall not attempt to summarize them further in this foreword. But some overview comments 
are in order. 

This study is fundamentally about what I would call the intellectual professionalization of 
intelligence analysis. It is about standards and practices and habits of mind. It is about 
inductive (evidence-based) analytical reasoning balanced against deductive (hypothesis- 
based and evidence tested) reasoning. It extols the value of truly scientific modes of thinking, 
including respect for the role of imagination and intuition, while warning against the pitfalls of 
"scientism," a false pretense to scientific standards or a scientific pose without a scientific 
performance. It talks about peer review and challenging assumptions and the need to build 
these therapeutic virtues into the analytical process. 

Mr. Cooper makes reference to the standards and practices of other professions with a high 
order of cerebral content, such as law and medicine. Other recognized authors, such as 
Stephen Marrin and Rob Johnston, have written persuasively on this theme. I am struck by 
how frequently Mr. Cooper — and others — refers to the example of medicine, especially 
internal medicine, which has much to offer our discipline. But I am not surprised. When I was 
very young in this business, I was fretting about its difficulties in the company of my uncle, 
an old and seasoned physician. He walked to his vast library and pulled out for me a volume, 
Clinical Judgment, by Alvan Feinstein, a work now often cited by intelligence reformers. I 
later asked my mother, my uncle's younger sister, what made Uncle Walt such a great 
doctor. Her answer: He always asks his patients at the beginning, "how do you feel?" and he 
never makes it home for dinner on time. The model of internal medicine is a great one for 
critical emulation by intelligence analysis: science, training, internship, expertise, 
experience, and then seasoned judgment, intuition, unstinting diligence, and valued second 
opinions. 



Most of what Mr. Cooper writes about concerns the intellectual internals of good intelligence 
analysis, i.e., standards, methods, the tool box of techniques, and the vital element of attitude 
toward understanding and knowledge building. With somewhat less emphasis but to good 
effect, he also addresses what might be called the environmental internals of the same: 
training, mentoring, incentives, management, and leadership. It is in this dimension that we 
must overcome the plague recognized by all informed critics, the tyranny of current 
intelligence, and restore the value of and resources for deep analysis. 

This leads to a consideration of the "externals" of good intelligence analysis. To wit: 

The full scope of analysis: This has to be appreciated for things to come out right. Analysis 
is not just what a hard-pressed analyst does at his desk. It is the whole process of cerebration 
about the mission and its product. This applies to not only the best answer to a current 
intelligence question on the table, but to establishing priorities, guiding collection, and, 
especially, to judging whether the best effort on the question of the day is good enough to 
support the weight of the situation and the policy decisions that have to be made. 

Money and people: There is no gainsaying that a lot of our failings after the Cold War are the 
fault of resource and personnel cuts while old and new and more equally competing priorities 
were proliferating. We've got to fortify the bench strength of intelligence analysis. The 
president has called for that. Without improved practices, however, new resources will be 
wasted. We press for improved practices; but they need more resources to be implemented 
effectively. 

External knowledge environments: Half a century ago, when the United States came to 
appreciate that it faced an enigmatic and dangerous challenge from the Soviet Union, it 
invested seriously in the building of knowledge environments on that subject, in the 
government, in think tanks, in academia, and in other venues. These external sources of 
expertise, corrective judgment, and early warning proved vital in keeping us on track with 
respect to the Soviet problem. We have yet to get serious about building such knowledge 
environments for the challenges of proliferation and, especially, concerning the great struggle 
within the world of Islam, from which the main threat of terrorism emerges. Related to this, 
Mr. Cooper's study properly places great importance on our improving exploitation of open 
sources. 

Information security regimes: We are talking here about a complicated domain from 
classification to recruitment and clearance systems. What we have is hostile to the task of 
developing a comprehensive, communitywide knowledge base and operational efficacy in 
the age of information and globalization. We need to be more open on a lot of things, 
especially where the original reason for secrecy perishes quickly and the value of openness 
is great (as during the Cold War in regard to Soviet strategic forces), and to tighten up on 
secrecy where it is vital, for example, in protecting true and valuable cover. 

One final — and perhaps most important — point: Mr. Cooper's study of intelligence analysis is 
shot through with a judgment that is shared by almost every serious professional I've heard 
from in recent years. And it applies to collection and other aspects of national intelligence as 
well. We cannot just rely on the new Director of National Intelligence (DNI) superstructure to 
put things right with our national intelligence effort. The problems and pathologies that inhibit 



our performance and the opportunities for radically improving that performance are to be 
found down in the bowels and plumbing of this largely dutiful ship we call the Intelligence 
Community, and that is where we must studiously, and with determination, concentrate our 
efforts and our money. 

— Fritz Ermarth 1 



1 Fritz Ermarth is a former chairman of the National Intelligence Council; he is now a security policy 
consultant. 
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Introduction 



As a result of a number of analytic projects for different intelligence agencies, a 
major focus of my work during the past several years has involved examining the 
practice of analysis within the US Intelligence Community. 1 This study was prompted 
by a growing conviction — shared by others, to be sure — that improving the analytic 
products delivered by Intelligence Community components had to begin with a 
critical and thorough appraisal of the way those products are created. A 
conversation with a physicist friend in 2002 had triggered thoughts on several basic 
differences between the practice of science and intelligence analysis. Shortly 
thereafter, an invitation to give a seminar on intelligence analysis at Stanford 
University led me to prepare a briefing entitled "Intelligence and Warning: Analytic 
Pathologies," which focused on a diagnosis of the problems highlighted by recent 
intelligence failures. 2 As Donald Stokes noted in his seminal book on science and 
technological innovation, Pasteur's Quadrant, "Pathologies have proved to be both 
a continuing source of insight into the system's normal functioning and a motive for 
extending basic knowledge." 3 

The Analytic Pathologies framework yields four insights that are crucial both to 
accurate diagnosis and to developing effective remedies. First, the framework 
enables analysts to identify individual analytic impediments and determine their 
sources. Second, it prompts analysts to detect the systemic pathologies that result 
from closely-coupled networks and to find the linkages among the individual 
impediments. Third, it demonstrates that each of these networks, and thus each 
systemic pathology, usually spans multiple levels within the hierarchy of the 
Intelligence Community. Fourth, the framework highlights the need to treat both the 
systemic pathologies and the individual impediments by focusing effective remedial 
measures on the right target and at the appropriate level. 

In response to presentations to community audiences, a number of senior 
intelligence officials subsequently recommended that I use the diagnostic framework 
of the briefing to develop corrective measures for the dysfunctional analysis 
practices identified there. I circulated the resulting draft for comment and was 
delighted to receive many useful suggestions, most of which have been incorporated 
in this version. 



1 Although this paper will use the common terminology of "Intelligence Community" (IC), it is worth 
noting that the agencies of which it is composed seldom exhibit the social cohesion or sense of purpose 
that a real community should. A more appropriate term might be "intelligence enterprise," which is 
defined in Webster's Third International edition as "a unit of economic or business organization or 
activity." 

2 The briefing was first presented in early November 2003 to a seminar at Stanford University's Center 
for International Security and Cooperation (CISAC) and was revised for a Potomac Institute seminar 
on the "Revolution in Intelligence Affairs" on 17 May 2004. It will be cited hereafter as "Analytic 
Pathologies Briefing." 

3 Donald E. Stokes, Pasteur's Quadrant: Basic Science and Technological Innovation. 



Several knowledgeable readers of the draft also raised the issue of the intended 
audience, strongly suggesting that this should be the senior decisionmakers, in both 
the Executive Branch and Congress, who could take action to implement the ideas 
it presented. They also pointedly recommended that the study be substantially 
condensed, as it was too long and "too rich" for that readership. That audience is, 
after all, composed of very busy people. 

From the beginning, however, I have intended this study to serve as a vehicle for an 
in-depth discussion of what I believe to be the real sources of the analytic 
pathologies identified in the briefing— the ingrained habits and practices of the 
Intelligence Community's analytic corps— and not the organizational structures and 
directive authorities that are the focus of most legislative and executive branch 
reformers. Thus, my intended audience has been the cadre of professional 
intelligence officers who are the makers and keepers of the analytic culture. Without 
their agreement on causes and corrective measures, I believe real transformation of 
intelligence analysis will not occur. 

Moreover, during the writing of this study, I was fortunate enough to serve on the 
selection panel for the inaugural Galileo Awards. 4 One of the winning papers 
focused on a similar issue — the appropriate audience for intelligence — and this 
reinforced my original inclination. 5 1 have decided, therefore, not to condense this 
study in an effort to fit the time constraints of very high-level readers. I hope, instead, 
that the summary that follows this introduction proves sufficiently interesting to tempt 
them to tackle the remainder of the study, where the logic chains that I believe are 
necessary to convince intelligence professionals of the correctness of the diagnosis 
and the appropriateness of the suggested remedies are laid out in detail. 



4 The Galileo Awards were an initiative of DCI George Tenet, who, in June 2004, invited members of 
the Intelligence Community to submit unclassified papers dealing with all aspects of the future of US 
intelligence. DCI Porter Goss presented the first awards in February 2005. 

5 David Rozak, et al., "Redefining the First Customer: Transforming Intelligence Through Peer- 
Reviewed Publications." 



Summary 



Observations 

A wide range of problems has contributed to the unease currently pervading the Intelligence 
Community; 1 a significant number of the most serious result from shortcomings in 
intelligence analysis rather than from defects in collection, organization, or management. 2 
The obvious and very public failures exemplified by the surprise attacks of 11 September 
2001 and by the flawed National Intelligence Estimate (NIE) of 2002 on Iraqi weapons of 
mass destruction (WMD) have resulted in a series of investigations and reports that have 
attempted to identify the causes of those failures and to recommend corrective actions. 3 
These recommendations have usually emphasized the need for significant modifications in 
the organizational structure of the Intelligence Community and for substantial enhancements 
of centralized authorities in order to better control and coordinate the priorities and funding 
of community entities. The Intelligence Reform and Terrorism Prevention Act (IRTPA) of 
2004, which created the office of Director of National Intelligence (DNI), was based on such 
foundations. 4 

The logic of this study differs from most of those recommendations with respect to both 
causes and corrective measures. The key observations in the original "Analytic Pathologies" 
briefing point in a fundamentally different direction for the root causes of the failures and for 
fixing the manifest problems. Most importantly, these observations lead to the conclusion 
that the serious shortcomings — with particular focus on analytic failures — stem from 
dysfunctional behaviors and practices within the individual agencies and are not likely to be 
remedied either by structural changes in the organization of the community as a whole or by 
increased authorities for centralized community managers. Those key observations, which 
follow, provide the conceptual foundation for this study. 

1. There has been a series of serious strategic intelligence failures. Intelligence support 
to military operations (SMO) has been reasonably successful in meeting the challenges on 
the tactical battlefield of locating, identifying, and targeting adversary units for main force 
engagements. Similar progress in supporting counterterrorism operations has been 
claimed. 5 At the same time, however, other military and national users have been far less 
well served by the Intelligence Community across a range of functions. There have been 
significant shortfalls in support to post-conflict security and stabilization operations and 



1 See The 9/11 Commission Report: Final Report of the National Commission on Terrorist Attacks Upon 
the United States (cited as the 9/11 Commission Report) and Report on the U.S. Intelligence 
Community's Prewar Intelligence Assessments on Iraq by the US Senate Select Committee on 
Intelligence, 7 July 2004 (hereinafter cited as SSCI Report). 

2 See Henry A. Kissinger, "Better Intelligence Reform," Washington Post, 16 August 2004: 17. 

3 For a review of the various commissions that have tackled intelligence reform, see Michael Warner 
and J. Kenneth McDonald, US Intelligence Community Reform Studies Since 1947. A detailed look at 
the work of one such recent commission is Loch K. Johnson, "The Aspin-Brown Intelligence Inquiry: 
Behind the Closed Doors of a Blue Ribbon Commission," Studies in Intelligence 48, no. 3 (2004): 1- 
20. Still, there is no guarantee that good intelligence will necessarily help decisionmakers reach good 
judgments or make good decisions, but poor intelligence can clearly corrupt good decision processes 
and amplify ill-advised tendencies in flawed processes. 

4 Intelligence Reform and Terrorism Prevention Act, PL 108^158, 2004 (hereinafter cited as IRTPA). 

5 See Testimony by Cofer Black, Coordinator for Counterterrorism, US Department of State, before the 
House International Relations Committee, 19 August 2004. 



reconstruction efforts in Iraq. Analytic support has also come up short both in accurately 
capturing adversary thinking and intentions and in providing intelligence that identifies and 
characterizes developing strategic challenges, such as WMD. 6 

Moreover, within the past decade and a half, a series of intelligence failures at the strategic 
level, including serious failures in operational and strategic warning, have highlighted real 
weaknesses at this level and undercut the confidence of principal national users in the 
community's capabilities against important intelligence targets. These failures include Iraqi 
WMD developments (1991 onward), the global black-market in WMD, strategic terrorism 
(beginning with the attack on the World Trade Center in 1 993), the North Korean nuclear 
program (1 994), the emergence of globally-networked Islamic fundamentalism (1 996 
onward), the Indian and Pakistani nuclear programs (1998), 7 the 9/11 attacks (2001), and 
Iran's WMD programs (2002). Similar failures, as well as an apparent inability to provide 
accurate assessments and estimates on other important issues, such as the nuclear forces 
and strategies of China and Russia, affect national users at the highest levels and outweigh 
any increases in effectiveness at the tactical level. 

Indeed, as a bottom-line assessment, this study contends that the Intelligence Community 
has been least successful in serving the key users and meeting the primary purposes for 
which the central intelligence coordinating apparatus was created under the National 
Security Act of 1947. 8 These principal officials are the president and his cadre of senior 
national security policymakers, not the departmental and battlefield users. As a senior 
intelligence official recently reminded us, those objectives were two-fold: not only to provide 
"strategic warning" in order to prevent another surprise such as Pearl Harbor, but also to help 
head off long-term challenges through a better understanding of the emerging strategic 
environment. 9 

2. These failures each have particular causes, but the numerous individual problems 
are interrelated. These failures did not have a single locus — they occurred in technical 
collection, human source reporting, and analysis, among other critical functions — but neither 
do they reflect a series of discrete, idiosyncratic problems. Instead, they resulted from deep- 
seated, closely-linked, interrelated "systemic pathologies" that have prevented the 
Intelligence Community from providing effective analytic support to national users, especially 
effective anticipatory intelligence and warning. 10 The Intelligence Community's complicated 



6 It appears, for example, that the intelligence needed to support the security and stabilization 
operations in Iraq with effective "cultural awareness" during the post-conflict "Phase IV" has been far 
less than adequate. See comments by senior military officers at a conference in Charlottesville, 
Virginia, sponsored by CIA's Center for the Study of Intelligence (CSI). Intelligence for a New Era in 
American Foreign Policy, (hereinafter cited as Charlottesville Conference Report), 3-5. 

7 Perhaps the more serious error in the case of the Indian-Pakistani nuclear tests was not the failure to 
predict the timing of the catalytic Indian test (which was really more a failure by policymakers); arguably, 
it was the failure to estimate correctly the scale and status of the Pakistani weapons program, including 
its links to the global WMD black market. 

8 Michael Warner, "Transformation and Intelligence Liaison," SAIS Review of International Affairs 
(hereinafter SAIS Review) 24, no. 1 (Winter-Spring 2004): 77-89. 

9 See Deborah Barger, "It is Time to Transform, Not Reform, U.S. Intelligence," SAIS Review24, no. 1 
(Winter-Spring 2004): 26-27. 

10 The systemic pathologies are discussed in detail in Chapter Three. 



organizational structure and the accreted practices of its analysts have combined to create 
what Charles Perrow calls "error-inducing systems" that cannot even recognize, much less 
correct their own errors. 11 

3. The Intelligence Community still relies on the same collection paradigm created for 
"denied areas." Remote technical collection and targeted human access were appropriate 
means of penetrating denied areas and obtaining critical intelligence against a 
bureaucratized, centralized, and rigid superpower adversary that exhibited strongly 
patterned behavior. The problem presented by many of the new threats, whether from 
transnational terrorist groups or from non-traditional nation-state adversaries, however, is 
not that of accessing denied areas but of penetrating "denied minds" — and not just those of 
a few recognized leaders, but of groups, social networks, and entire cultures. Unfortunately, 
information for intelligence is still treated within the old "hierarchy of privilege" that 
emphasized "secrets" and was more appropriate for a bureaucratized superpower adversary 
who threatened us with large military forces and advanced weapons systems. 12 Without 
refocusing its energies, the Intelligence Community will continue to do better against things 
than against people. 

4. Analytic methods also have not been updated from those used to fight the Cold War. 

There were intelligence failures during the Cold War, but the United States and its allies 
managed to stay on top of the challenge presented by our principal adversary. A relatively 
stable threat (and consistent single target) allowed the Intelligence Community to foster in- 
depth expertise by exploiting a very dense information environment, much of which the 
opponent himself created. That "Industrial Age" intelligence production model — organized for 
efficiency in high-volume operations and fed by large-scale, focused, multiple-source 
collection efforts conducted mostly with episodic "snapshot" remote systems that were very 
good at big fixed targets — built a solid foundation of evidence. This knowledge base allowed 
analysts to cross-check and corroborate individual pieces of evidence, make judgments 
consistent with the highest professional standards, and appreciate and communicate any 
uncertainties (both in evidence and inference) to users. In particular, this dense information 
fabric allowed analysts to place sensitive intelligence gathered from human sources or by 
technical means within a stable context that enabled confirmation or disconfirmation of 
individual reports. As national security challenges evolved during the years following the 
collapse of the Soviet Union, however, continued reliance on the Cold War intelligence 
paradigm permitted serious analytic shortfalls to develop. 

5. The Intelligence Community presently lacks many of the scientific community's 
self-correcting features. Among the most significant of these features are the creative 
tension between "evidence-based" experimentalists and hypothesis-based theoreticians, a 
strong tradition of "investigator-initiated" research, real "horizontal" peer review, and "proof" 
by independent replication. 13 Moreover, neither the community as a whole nor its individual 



11 Charles Perrow, as cited in Robert Jervis, "What's Wrong with the Intelligence Process?" 
International Journal of Intelligence and Counterintelligence 1 , no. 1 (1986): 41 . See also Charles 
Perrow, Normal Accidents: Living with High-Risk Technologies. 

12 Fulton Armstrong, "Ways to Make Analysis Relevant But Not Prescriptive," Studies in Intelligence 46, 
no. 3 (2002): 20. 



analysts usually possess the ingrained habits of systematic self-examination, including 
conducting "after action reviews" as part of a continual lessons-learned process, necessary 
to appreciate the changes required to fix existing problems or to address new challenges. 14 

6. Intelligence analysis remains a "craft culture," operating within a guild structure 
and relying on an apprenticeship model that it cannot sustain.' 15 Like a guild, each 
intelligence discipline recruits its own members, trains them in its particular craft, and 
inculcates in them its rituals and arcana. These guilds cooperate, but they remain distinct 
entities. Such a culture builds pragmatically on practices that were successful in the past, but 
it lacks the strong formal epistemology of a true discipline and remains reliant on the 
transmission, often implicit, of expertise and domain knowledge from experts to novices. 
Unfortunately, the US Intelligence Community has too few experts — either analytic "masters" 
or journeymen — left in the ranks of working analysts to properly instruct and mentor the new 
apprentices in either practice or values. 



Conclusions 

The Intelligence Community is not normally self-reflective and usually avoids deep self- 
examination, but recognition and acceptance of the seriousness of its problems by all levels 
of the community is a necessary prerequisite for true change, including significant 
modifications to current organizational cultures and ethos. Agreement on the basic diagnosis 
must, therefore, precede detailed propositions about effective remedies. I suggest that the 
following six premises, first articulated in the "Analytic Pathologies" briefing, summarize the 
most important conclusions to be drawn from the preceding discussion of the current 
enfeebled state of the Intelligence Community. 

1. The dysfunctional practices and processes that have evolved within the culture of 
intelligence analysis go well beyond the classic impediments highlighted by Richards 
Heuer in The Psychology of Intelligence Analysis. 16 A more effective analytic paradigm 
must be built that incorporates the best analytic methods from modern cognitive science and 
employs useful and easily usable supporting tools to overcome these impediments and 
prevent them from combining into systemic pathologies. 



13 "Evidenced-based" analysis is essentially inductive; "hypothesis-based" is deductive; they should be 
seen as complementary approaches, not competitors for ownership of the analytic process. 

14 For an exception, see John Bodnar, Warning Analysis for the Information Age: Rethinking the 
Intelligence Process. In fact, both the Joint Military Intelligence College (JMIC) and the Center for the 
Study of Intelligence have programs to create a discipline of intelligence by bringing together 
intelligence theory and practice. Regrettably, the results of these efforts have not yet penetrated the 
mainline analytic units. 

15 In fact, the analytic community self-consciously characterizes its practices and procedures as 
"tradecraft." 

16 Richards J. Heuer Jr., The Psychology of Intelligence Analysis. Building on the work on cognitive 
impediments to human judgment and decisionmaking of Daniel Kahneman, Amos Tversky, and others, 
in addition to his own long experience as a senior intelligence analyst, Heuer highlighted many 
psychological hindrances to making accurate judgments by individuals and small-groups. 



2. More corrosively, the individual impediments form interrelated, tightly-linked, 
amplifying networks that result in extremely dysfunctional analytic pathologies and 
pervasive failure. A thorough reconceptualization of the overall analysis process itself is 
needed. The new approach would incorporate a better connected, more interactive, and 
more collaborative series of networks of intelligence producers and users. In addition, it must 
be designed to detect and correct errors within routine procedures, instead of leaving them 
to be found by post-dissemination review. 

3. The new problems and circumstances call for fundamentally different approaches 
in both collection and analysis, as well as in the processing and dissemination 
practices and procedures that support them. It is clear that serious problems in the 
existing organizational structure of the Intelligence Community are reflected in poor 
prioritization, direction, and coordination of critical collection and analysis activities. 
However, many problems that are more fundamental and deep-seated exist inside the 
organizational "boxes" and within the component elements of the intelligence agencies 
themselves. Fixing these — dysfunctional processes, ineffective methods, and ingrained 
cultures — is not solely a matter of increased authorities, tighter budgetary control, or better 
management. A strategic vision that addresses the systemic pathologies, leadership that 
understands how key functions ought to be improved, and a sustained long-term 
commitment to rebuilding professional expertise and ethos will be essential. 

4. Accurate diagnosis of the root causes of problems "inside the boxes" is required; 
otherwise remedies will be merely "band-aids." For example, the analytic problems occur 
at and among four organizational levels: 1) individual analysts; 2) analytic units, including 
their processes, practices, and cultures; 3) the individual intelligence agencies; and 4) the 
overall national security apparatus, which includes the entire Intelligence Community in 
addition to the executive bodies responsible for making policy. Solving problems at all four 
of these interlocking levels requires an integrated attack that includes solutions addressed to 
the right level and tailored for each problem element. 

5. The Intelligence Community must bring more perspectives to bear on its work and 
create more effective "proof" and validation methods in constructing its knowledge. 

It should, in particular, adopt proven practices from science, law, and medicine, including 
more open communication and self-reflection. 

6. Whatever the details of structures or authorities, the new Director of National 
Intelligence (DNI) leadership must assure that the corrective measures are 
implemented within each agency and across the community. Moreover, all this should 
be done in the knowledge that change will be continual and that there will be no static resting 
place where the "right" solutions have been found; organizational structures and processes 
must be designed to evolve with and adapt to that realization. 17 



Recommendations 

Curing the flaws in intelligence analysis will require a sustained emphasis on rebuilding 
analytic capabilities, refocusing on human cognitive strengths enhanced by innovative 
support tools, and restoring professional standards and ethos among the analysts 



themselves. Most of the recent reform recommendations notwithstanding, more guidelines 
and tighter management oversight are no substitute for analytic expertise, deep 
understanding, and self-imposed professional discipline — all achieved not only by formal 
education and training, but also through assimilation from following experienced mentors. 
Moreover, neither curiosity nor expertise on the part of the individual analysts can be restored 
by directives from the top; they must come from an appropriate recruiting profile, effective 
training, continual mentoring at all levels, time to learn and practice the craft of analysis — 
both individually and collaboratively — and constraining the "tyranny of the taskings" that 
prevents analysts from exercising curiosity and pondering more than the obvious answer. 18 

To ensure that the Intelligence Community can provide more effective capabilities to meet 
the increasingly complex challenges of 21 st-century security issues, this study recommends 
rebuilding the overall paradigm of intelligence analysis from its foundations. The essential 
components of this effort are: 

1 . A revamped analytic process; 

2. An entirely revised process for recruiting, educating, training, and ensuring the 
professional development of analysts (including the essential aspect of mentoring); 

3. Effective mechanisms for interactions between intelligence analysts and users; 

4. A proper process for "proof," validation, and review of analytic products and 
services; 

5. An institutionalized lessons-learned process; 

6. Meaningful processes for collaboration within the Intelligence Community. 

Furthermore, although implementing each of these processes separately would produce 
significant improvements in the quality of analysis, a more effective approach would be to 
mount a broad-gauged, systematic, and integrated effort to deal with the entire analysis 
process. 



17 A medical analogy might make this argument clearer. Although a low-cholesterol diet, proper 
exercise, routine physicals, a low dose of aspirin, and moderate intake of alcohol may be useful over 
the long-term for preventing heart disease, patients in acute cardiac distress require more forceful 
intervention to save them. The measures listed above would have been useful before the attack, and 
they may be appropriate after recovery, but they are not effective during an acute crisis or in the 
immediate aftermath, when patients must be kept under observation to be certain they are "taking their 
medicine." 

18 Professor Jeffrey Pfeffer of the Graduate School of Business at Stanford University is one of several 
commentators who have emphasized the importance of "slack" to enable collaboration and collective 
efforts — including discussion, review and comment, professional development, and service to the 
"community of practice," as well as pursuing the scent of curiosity. 
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A Complex Adaptive System 

With its fifteen diverse agencies and its wide 
range of functional responsibilities, the Intel- 
ligence Community presents a very compli- 
cated set of organizational arrangements. 
Thinking of it in terms of traditional organiza- 
tional analysis or systems engineering meth- 
ods in an effort to explain its working does 
not suffice because it far more resembles a 
living ecology with a complex web of many 
interacting entities, dynamic relationships, 
non-linear feedback loops (often only par- 
tially recognized), and specific functional 
niches that reflect momentarily successful 
adaptations to the environment. 1 These 
complex interrelationships among its com- 
ponents create dynamic adaptations to 
changing conditions and pressures and 
make the Intelligence Community especially 
difficult to understand. 2 In fact, it is an exem- 
plar, even if not a healthy one, of a truly com- 
plex adaptive system. 

During the Cold War, proportionately more 
resources supporting a larger cadre of expe- 
rienced analysts devoted to a simpler and 
relatively static priority target, as well as a 
broad array of established sources, dis- 
guised many of the Intelligence Commu- 
nity's dysfunctional aspects and growing 
internal problems. The community's loosely 
federated structure and complicated, if not 
Byzantine, processes had previously 
appeared tolerable, even if not fully success- 
ful, because making changes appeared to 
present a greater risk. 3 In the face of a dras- 



tically changed security environment, how- 
ever, it is exactly the combination of 
complexity and opaqueness that has 
masked the increasingly dysfunctional mis- 
alignment of "dinosaur" analytic processes 
and methodologies from earlier recognition 
by both analysts and consumers of intelli- 
gence, much less by outsiders. 4 

Even for insiders, the workings of the Intelli- 
gence Community are difficult to understand 
because, as a rule, its members are not 
deeply self-reflective about its practices and 
processes. For outsiders, however, these 
difficulties are magnified by the community's 
compartmentation, security restrictions, and 
intrinsic opaqueness. That is why applying 
traditional organizational analysis that con- 
centrates on structure is doomed to failure; 
understanding these complex adaptive sys- 
tems requires more synthesis than tradi- 
tional "reductionist" analysis. 5 In this case, 
moreover, it is a complex adaptive system 
that, insulated by security barriers, has man- 
aged to ignore and — probably because of its 
centralized direction, however imperfect — 
suppress important external signs of change 
and to amplify self-protective internal sig- 
nals, which often reflect strongly ingrained 
cultural preferences. 

The results of the Intelligence Community's 
failure to recognize the increasing dysfunc- 
tion were both paradoxical and unfortunate. 
They were paradoxical because — although 
it has been accused of not adapting to dra- 
matically changed conditions — the commu- 



The Intelligence 
Community is an 
exemplar, even if 
not a healthy 
one, of a truly 
complex adap- 
tive system. 



1 A feedback loop, in systems analysis, is a relationship in which information about the response of the system to stimuli 
is used to modify the input signal (see "Feedback," Phncipia Cybernetica Web). A non-linear loop is one that creates 
non-proportional responses to stimuli. 

2 See Peter M. Senge, The Fifth Discipline: The Art & Practice of the Learning Organization. Senge is the founder of 
the Organizational Learning Laboratory at MIT. 

3 The pressures of the Manichean confrontation with the Soviet Union tempered enthusiasm for drastic and disruptive 
changes. These might have improved effectiveness, but they would also have provoked bureaucratic and 
congressional battles over power and jurisdiction. 

4 After all, the dinosaurs were superbly adapted to their environment; even if they perceived the signals of change, they 
became extinct because they could not adapt to unfamiliar environmental conditions. 

5 An appreciation of the distinction between a complicated system and one that is complex and adaptive is important 
for accurate diagnosis and effective solutions. A hallmark of complex adaptive systems is that they produce "emergent 
behavior," which cannot be prediced by analysis of their component elements or structure. 
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only to locate the 
level at which ob- 
vious symptoms 
occur, but also 
the level at which 
problems can be 
solved. 



nity adapted all too well. And they were 
unfortunate because the pressures to which 
it did adapt flowed from misperceptions 
inside and outside the Intelligence Commu- 
nity engendered by the collapse of the 
Soviet Union: that there would be no signifi- 
cant challenges to American interests; that 
the end of the Cold War reduced the need 
for a "national security state"; that there 
should be a substantial "peace dividend," a 
large part of which would be paid by the 
Intelligence Community. The community's 
adaptive processes did accommodate 
these changes internally — especially the 
need to "survive" the huge budget cuts and 
to become relevant to the articulated needs 
of the paying customers. 

However, these internal pressures out- 
weighed the huge new challenges emerging 
in the external security environment. 
Responding to these would demand new 
expertise and a new knowledge base, along 
with appropriate methods, tools, and per- 
spectives — all of which required more 
resources, focused leadership, and strong 
commitment, which was not there. As a 
result, the community fostered a series of 
processes that were increasingly mal- 
adapted to needs emerging in the new geo- 
strategic environment. By responding to the 
wrong signals, it created Perrow's "error- 
inducing systems." 6 



Relating Structure and Process 

Unfortunately, most Intelligence Community 
reform proposals concentrate on changes in 
structure and in directive and managerial 
authorities. Analytic problems, however, 



actually take place not just at the level of the 
community as a whole, but at four distinct 
levels, as well as in the complex interrela- 
tionships, both vertical and horizontal, 
among them. 7 Thus, it is important not only 
to locate the level at which the obvious 
symptoms appear, but also the level at 
which the problem can be solved. In this 
way, the root causes of failure can be iden- 
tified and appropriate and effective correc- 
tive measures taken. 

The National Security Community. The 

relevant entities include the National Secu- 
rity Council (NSC), the Office of the Director 
of National Intelligence (ODNI), and the 
national policymaking and operational ele- 
ments in the Department of State and the 
Department of Defense. 8 Among the fail- 
ures at this level can be misdirected priori- 
ties and misallocation of resources; poor 
communication and coordination; and 
inconsistent apportionment of authority, 
responsibility, and capability among the 
main entities. Such failures flow downward 
and can easily percolate throughout the 
subordinate organizations. 

For the Intelligence Community, a particular 
problem at this level may involve its relation- 
ships with top-level users, especially man- 
aging their expectations. On the one hand, 
for example, the Intelligence Community 
often demonstrates an inability or unwilling- 
ness to say "no" to consumer requests, 
which leads to additional priority taskings 
without concomitant resources or relief from 
other ongoing activities. Similarly, the Intelli- 
gence Community often conveys an illusion 
of omniscience that fails to make clear its 
state of knowledge on an issue, the underly- 



6 See Perrow, Normal Accidents. 

7 The briefing on "Analytic Pathologies" graphically illustrates the multi-level interplay of these problems. See 
Appendix A for a summary. 

8 At this level, for the Intelligence Community, it is the ODNI and the Intelligence Community elements that are 
responsible for critical functions— collection, analysis, special activities, and community management— that interact 
directly with senior principals. With a DNI and an ODNI organization in place, these relationships are likely to become 
even more complicated. 



10 



Chapter One: Making Sense of the US Intelligence Community 



ing quality of the intelligence, or the degree 
of uncertainty — all of which can leave the 
Intelligence Community seemingly respon- 
sible for decisions taken on the basis of "bad 
intelligence." 

The Intelligence Community. This level 
currently includes the fifteen component 
intelligence agencies. Failures at this level 
can include misdirected priorities and bud- 
getary allocations within the Intelligence 
Community; lack of effective procedures 
and oversight of them among component 
agencies; poor communication and coordi- 
nation among agencies; a lack of enforce- 
able quality-control processes; toleration of 
substandard performance by individual 
agencies; poor communitywide technical 
standards and infrastructure that hinder 
information sharing; and poor management 
and oversight of security procedures that 
impede effective performance. Errors at this 
level also encompass failures by groups or 
individuals to make critical decisions, to 
exercise appropriate authority, or to take 
responsibility for gross errors that should be 
worthy of sanction or dismissal. 9 

The Individual Analytic Units and Orga- 
nizations. It is essential to appreciate the 
importance of particular analytic environ- 
ments within specific sub-organizations — 
an office within the CIA's Directorate of 
Intelligence, for example. It is these entities, 
rather than the organization as a whole, that 
create the work processes and practices 
that form the immediate cultural matrix for 
an analyst's behavior. 10 Failures at this level 
can include dysfunctional organizational 
processes, practices, and cultures that 
inhibit effective analysis by individuals and 
sub-units; management attitudes and direc- 
tives that stress parochial agency objec- 



tives; toleration of poor performance; 
excessive compartmentation and special 
security procedures that erect barriers to 
effective execution; poor prioritization and 
assignment of workflow; inability to create 
and protect "slack" and conceptual space 
for intellectual discovery; ineffective recruit- 
ment and training; maintaining stand-alone 
information and analysis infrastructures, 
including ineffective support for individual 
analysts; poor direction and management of 
the analytic process; and, simply, ineffective 
management of the analytic cadre. This is 
probably the most important level for creat- 
ing consistently high-quality analysis 
because of its impact on the analytic envi- 
ronment, on the selection of methods and 
processes, and on the work life of individual 
analysts. Errors at this level are perhaps the 
most pernicious, however, and they have 
been widespread and persistent. 

Individual Analysts. Failures at this level 
can include poor performance due to lack of 
ability, lack of domain knowledge, lack of 
process expertise, poor social network con- 
tacts, or ineffective training; pressures to 
favor product over knowledge; lack of time; 
being too busy and too focused to maintain 
peripheral vision and curiosity, even on high 
priority work; failure to cooperate and col- 
laborate with others; lack of suitable tools 
and support; misguided incentives and 
rewards; and an organizational culture and 
work practices that tolerate second-rate 
analysis. 

To illustrate the impact of this multi-level 
hierarchy and underscore the importance of 
correctly identifying the locations of caus- 
ative factors in analytic errors, for example, 
consider the case of an analyst who fails to 
interpret correctly the evidence pertinent to 



9 See Statement by Admiral David Jeremiah (USN, ret.), Press Conference, CIA Headquarters, 2 Jun 1998, for a 
suggestion that failures by senior managers to make key decisions had been an important factor in the CIA's failure 
to warn of an impending Indian nuclear test. (The subject was the "Jeremiah Report" on the 1 998 Indian nuclear test.) 

10 See Karl E. Weick, Sensemaking in Organizations. 
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a task and draws a wrong conclusion. At 
first glance, the obvious approach should be 
to focus corrective actions on the analyst: 
what caused the failure, and what are the 
appropriate remedies? Simple incompe- 
tence, a rush to complete the assignment, a 
lack of domain knowledge needed to recog- 
nize critical linkages, or a failure to employ 
appropriate methods could all be causative 
factors. At this level, the obvious remedies 
to these problems are better screening, 
training, and mentoring. 

It could be, however, that the problem lies 
with the analytic unit, its work processes, 
and its management: the tasking was high 
priority, and this analyst, whose expertise is 
on another subject, was the only one avail- 
able; appropriate tools and methods were 
not provided; training in relevant domain 
knowledge or on effective new tools had 
been postponed due to production pres- 
sures; or, given the production cycle, the 
analyst lacked sufficient time to search for 
all the relevant evidence. The problem could 
reside even farther up the hierarchy, among 
the agencies of the Intelligence Community: 
key data from another agency was not 
made available, due to compartmentation 
restrictions or because incompatible infor- 
mation infrastructures prevented the analyst 
from easily searching another agency's 
holdings. Finally, the failure could actually 
reside at the topmost level, with community 
management: this account was given such 
low priority that no collection resources had 
been assigned to gather information or to 
provide consistent analytic coverage or, 
because of the thinness of the evidence 



base, the inability to answer the question 
was not made clear to the requester at the 
start. 

However, it is exactly here that the "5 Whys 
Approach" of the Quality Movement proves 
its value. 11 Applying this approach, which 
features a progressively deeper, recursive 
search, forces the investigator to trace a 
causative factor to its source. 12 Assume 
that, in this example, it is a lack of domain 
knowledge. 

Why was an analyst not fully knowl- 
edgeable in the domain working that 
account? 

She was covering for the lead analyst, 
who is away on temporary duty (TDY). 

Why did the analytic manager assign 
that analyst to the task? 

She was the only one available. 

Why was the analyst not fully knowl- 
edgeable on her backup account? 

She is an apprentice analyst with only 
a short time on the account and inade- 
quate mentoring. Her training had 
been postponed due to scheduling. 
She didn't have time to be curious and 
follow the information scent. She could 
not access the lead analyst's "shoe- 
box." 13 

Why couldn't she access the shoebox 
of the lead analyst? 



11 The Quality Movement took root in the United States during the 1990s, when US auto manufacturers were 
challenged by the emergence of higher quality Japanese automobiles made by automakers who had adopted the 
principles of two US engineers, W. Edwards Deming and Joseph Juran. The principles provide a systematic set of 
processes and metrics for improving the quality of manufacturing processes. 

12 A recursive search is one in which successive searches build on the results of earlier searches to refine the answers 
returned. (See National Institute of Standards and Technology, Dictionary of Algorithms and Data Structures.) 

13 Although seldom used today, many analysts once referred to the personal files where they stored such items as 
the results of research as "shoeboxes." It is used here to emphasize the particularity of the methods employed by 
analysts. 
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It is his personal collection of tentative 
hypotheses and uncorrelated data 
kept as a personal Word file and is not 
in an accessible database. The shoe- 
box is actually a pile of paper put in 
temporary storage when the lead ana- 
lyst went on TDY. 

Why is the lead analyst unwilling to 
share his shoebox? 

Why is there no accessible collabora- 
tive system for sharing shoeboxes? 

The questions would continue through as 
many rounds as the questioner needed to 
satisfy himself that he had found the root 
cause. 

Although the previously cited reports on 
intelligence failures usually point to organi- 
zational stove-piping and technical short- 
comings as the most important contributors 
to failures in collaboration, the sources of 
such failure are actually more widespread 
and complex — and more frequently reflect 
shortcomings in work practices and pro- 
cesses, organizational culture, and social 
networks. 14 In addition, the proposed solu- 
tions that focus on structures and authorities 
disregard the critical interrelationship 
between structure and processes and 
ignore as well the importance of organiza- 
tional culture on institutional effectiveness. 
As Stephen Marrin, among others, has 
noted: 

Structure and process must work 
together in a complementary fashion, 
and structural changes alone without 
corresponding changes to existing 
processes would simplify the workings 



of the Intelligence Community in some 
ways, but cause greater complexity in 
others. 15 

The significant structural reforms legislated 
in 2004 will also entail substantial short-term 
transition costs to effectiveness as new 
organizational arrangements are imple- 
mented, processes are developed, and out- 
moded roles and systems are replaced. The 
really difficult task will be to redesign the 
processes, so that they are consistent and 
complementary to the structural changes 
that are being made. 



The Analysis Phase-Space 

At a basic level, incorrect diagnoses of the 
causes of analytic failures probably arise 
from not recognizing the variety and com- 
plexity of the roles, missions, and tasks that 
confront analysts. This diversity results in a 
complex phase-space, illustrated below, 
that contains a significant number of dis- 
crete analytic regions. These certainly can- 
not be treated as though their perspectives 
and needs were homogeneous or even sim- 
ilar. The tasks required of a signals intelli- 
gence analyst attempting to locate a 
terrorist's cell-phone call are fundamentally 
different from those of an all-source analyst 
drafting an NIE on Chinese strategic nuclear 
doctrine. Therefore, because intelligence 
collection and analysis are not based either 
on a suite of all-purpose tools or on fungible 
human expertise that can be instantly swiv- 
eled to focus effectively on a different set of 
problems, this phase-space also implies the 
need for a similar diversity of analytic pro- 
cesses, methods, knowledge bases, and 
expertise. 
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14 Technical systems and infrastructures enabling collaboration are important, but they are only a small part of the 
solution to fostering effective collaboration. For more on this topic, see discussion beginning on page 57. 

15 Stephen Marrin, in a review of William E. Odom, "Fixing Intelligence: For a More Secure America," Political Science 
Quarterly, 119, no. 2 (Summer 2004): 363. 
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A phase-space is a conceptual tool used by physicists to 
represent the abstract set of all potential dynamic values of a system that can be 
produced by every combination of system variables possible in each dimension. The 
relatively simple, 3-valued phase-space for analysis shown above includes 
dimensions for different domains and accounts, types of products and services, and 
sources of intelligence. 



Differentiating Intelligence Roles 

Moreover, given this diverse phase-space, 
conflating three distinct roles played by all- 
source intelligence adds to the underlying 
confusion over intelligence missions and 
functions, the priorities among them, their 
requirements, and the capabilities needed 
to make each effective. The traditional 
assumption that there were only two sets of 
intelligence consumers, each with distinct 
mission needs, often led to contraposing 



support to military operations, which was 
assumed to be tactical in focus, and national 
user support, which was assumed to 
demand deep analysis. In reality, meeting 
the disparate needs of the users intelligence 
must serve requires recognizing three dis- 
tinct roles for all-source intelligence. 16 Two 
of them, Support to Military Operations 
(SMO) and Support to Policy Operations 
(SPO), focus primarily on issues needing 
immediate information capabilities to assist 
decisionmaking on current operations. 
Although SMO and SPO issues are of inter- 
est to both national and departmental users, 
the third role, Warning and Estimative Intel- 
ligence (WEI), largely emphasizes issues 
that are almost exclusively the province of 
national users and usually take place over 
longer time horizons. 17 

In all cases, however, although it still uses 
the term "support," the Intelligence Commu- 
nity must move beyond the notion that it is 
segregated from the rest of the national 
security community and that it merely pro- 
vides apolitical information to decisionmak- 
ers. Intelligence has now become an 
integral element of both the policy and mili- 
tary operational processes; and the success 
or failure of its judgments can have the most 
significant consequences in both 
domains. 18 Increasingly-integrated military 
operations, in which intelligence directly 
drives operations, and command centers in 
which intelligence personnel are fully inte- 
grated, are tangible evidence of such 
changes. As a result, it is important that 
intelligence appreciate not only the central- 



16 It is important to recognize that these regions have fuzzy boundaries, overlap to some degree, and are not totally 
distinct. 

17 The intelligence role that often leads to confusion over appropriate categorization is warning, and especially the 
tactical warning component. Because warning is intimately connected to a decision on a responsive action, it is 
sometime mistakenly considered to be a decision-support activity; in reality, it is more appropriately seen as a part of 
the informative function that assists policymakers in thinking about issues before they occur, helping to create 
coherent, contextualized reference frames. Moreover, because tactical warning is tactical, it is often forgotten that it 
is of principal concern to high-level strategic users because it almost always involves activities that could have the 
most serious political and strategic consequences. Thus, these three roles cover two distinct functions: SMO and 
SPO emphasize situational awareness and immediate decision support, while WEI focuses on anticipation of future 
circumstances. 
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ity of its role, but also the increased obliga- 
tions and responsibilities that such a role 
brings. 19 

Support to Military Operations (SMO): 

This traditional intelligence role has usually 
focused on assisting current military opera- 
tions. Much of this information concerns cur- 
rent numbers, locations, and activities of 
hostile units, and other information 
addresses significant elements of the phys- 
ical environment in which military forces are 
operating. 20 Other military users need quite 
specific current data on subtle technical 
characteristics of adversarial equipment 
and forces to serve, for example, as target- 
ing signatures or to support electronic war- 
fare (EW) activities. Regardless of type, 
intelligence supporting operating forces 
demands extraordinary accuracy, precision, 
and timeliness to ensure that it is immedi- 
ately "actionable" under conditions that are 
highly stressful and potentially lethal. 21 

Increasingly, however, military operators 
have other operational intelligence needs, 
such as support for information operations 
and for security and stabilization in Iraq. To 
prosecute these missions successfully, the 
military now also needs far more cultural 
awareness and timely accurate information 
on adversary thinking, motivations, and 
intentions. 

Support to Policy Operations (SPO): 

Making explicit that this is a distinct role 
emphasizes the importance of intelligence 
to daily policymaking across the entire spec- 
trum of national security concerns; it is the 
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"national user" cognate of SMO. SPO pro- 
vides policymakers and senior officials 
(importantly including senior civilian 
defense officials, combatant commanders, 
and other military officers) with indispens- 
able situational awareness, including impor- 
tant background information, to assist them 
in executing and overseeing ongoing policy 
activities and in planning and framing policy 
initiatives. As it is as intensely focused on 
providing actionable information, it is as 
heavily oriented as SMO to current intelli- 
gence and reporting. However, SPO differs 
from SMO somewhat in content and priori- 
ties in that it has always included a greater 
proportion of less quantifiable, softer infor- 
mation, such as political and economic 



18 1 am grateful to Dr. Russell Swenson of the Joint Military Intelligence College for persuading me to sharpen this 
point. See Russell G. Swenson, with Susana C. Lemozy, "Politicization and Persuasion: Balancing Evolution and 
Devolution in Strategic Intelligence," unpublished manuscript. When the CIA was created, expectations about 
intelligence capabilities and its role were significantly different than they are today. At the policy level as well, there is 
now an expectation that intelligence will be available to guide policy creation and inform course changes if necessary. 

19 A valuable guide to appropriate comportment in these circumstances is Herbert Goldhamer's The Adviser. 

20 The US Army, which has extensive doctrine on operations, calls this intelligence preparation of the battlefield (IPB). 
This includes specific information on mission, enemy, time, terrain, and troops available (METT-T). 

21 A critical example is the need for technical details, so that enemy weapons, such as improvised explosive devices 
(lEDs), can be countered. 
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trends in major countries and groups and 
assessments of foreign leaders and their 
intentions. 

Warning and Estimative Intelligence 
(WEI): Mary McCarthy, a former National 
Intelligence Officer (NIO) for Warning, com- 
mented on the recommendations of a DCI- 
chartered study conducted in 1992: 

According to that ten-member panel of 
highly respected intelligence and pol- 
icy veterans, providing policymakers 
with persuasive and timely intelligence 
warning is the most important service 
the Intelligence Community can per- 
form for the security of the United 
States. 22 

McCarthy defines warning as "a process of 
communicating judgments about threats to 
US security or policy interests to decision- 
makers." 23 Thus, warning provides vital sup- 
port to "national users" in their principal stra- 
tegic missions — understanding the complex 
geostrategic environment, fostering vision 
of objectives, assessing alternatives and 
determining strategy, and protecting against 
consequential surprise; most importantly, 
when done properly, warning is forward 
looking and anticipatory. 24 

Warning is sometimes thought to be merely 
alerting decisionmakers to immediately 
threatening activities, but, in reality, it is a far 
more complex function and actually 
addresses two very different kinds of prob- 
lems. One type of warning is concerned with 



monitoring activities previously recognized 
as potentially dangerous, such as a hostile 
missile launch, and cueing appropriate 
responses. The second type is a discovery 
function that assists decisionmakers in 
identifying those situations and activities 
whose consequences could have significant 
(and usually adverse) effects — and which 
may not necessarily be obvious. When per- 
formed effectively, a warning process pro- 
vides decisionmakers with an anticipatory 
sensitization that allows them to think 
through, in a disciplined way, the responses 
they might someday be obliged to make in 
haste. Assessments and estimates, on the 
other hand, also are usually forward looking, 
but they are designed to be informative 
rather than part of a process closely tied to 
triggering contingent responses. 

Further complicating the matter is that both 
types of warning also operate over three dif- 
ferent horizons. Strategic warning has 
always been understood as looking out 
toward the distant future; it is intended to 
recognize that a possible threat may be 
looming — even if it is not imminent — and to 
provide time to take appropriate preparatory 
actions, including policies and actions that 
might prevent the threat from eventuating. 25 
Operational warning also looks out in order 
to identify the characteristics of the threat 
(the likely and particular methods of attack), 
so that offsetting contingency plans and 
actions can be prepared. From this detailed 
understanding of enemy intentions, capabil- 
ities, and concepts, operational warning 
also serves to identify indicators that an 



22 Mary McCarthy, "The National Warning System: Striving for an Elusive Goal," Defense Intelligence Journal 3 

(1 994): 5. Warning is considered the classic "strategic intelligence" role and was the principal reason for the creation 
of the CIA. 

23 Ibid. 

24 There have always been terminological problems associated with the word "strategic." During the Cold War, users 
of the word often conflated level of analysis (global and synoptic), time horizon (forward-looking), and magnitude of 
the stakes (very large), with instrumentality (nuclear) and distance (intercontinental). 

25 To some degree, these terms have always been confusing because they described two very different types of 
problems. Strategic, operational, and tactical warning related to surprise nuclear attack were very patterned, but 
focused on two distinct problems: surprise attack executed by known forces and surprises that were truly 
unanticipated. 
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attack is in preparation. Finally, tactical 
warning is the immediate alerting function 
that a specific (with respect to time, place, or 
character) hostile activity has begun or is 
about to begin. 

An important but often overlooked element 
of warning over all three horizons is the key 
role played by negative evidence, which can 
help confirm that potentially threatening 
activities are not occurring and prevent 
costly and potentially consequential 
responses from being taken or scarce 
resources from being squandered. 26 During 
the confrontation between the United States 
and the Soviet Union, and, in particular, dur- 
ing periods of high tension between them, 
one of the most important functions of warn- 
ing was to inform the leaders that, "Today is 
not the day." 

Both the warning and estimative functions 
are designed to focus more on informing 
decisionmaking with context and long-term 
implications than with supporting ongoing 
activities. The preparation of assessments 
and estimates, as well as development of 
warning indicators, has more to do with 
analysis and judgmentthan with reporting; it 
demands deep expertise as well as an abil- 
ity to place knowledge of the subject in 
broad context. These important functions 
serve the entire national security commu- 
nity. 27 

Although warning is often misconstrued as a 
current intelligence problem, even tactical 
warning of specific targets, times, and 
means must build on this deeper foundation 



of pre-emptive analysis of threats and 
responses if it is to be effective. During the 
Cold War, recognizing that we were 
engaged in a long-term competition, we 
were prepared to adjust our intelligence pri- 
orities so that analysts could provide 
assessments of future capabilities and indi- 
cations of intentions, even though the day- 
to-day threats were most grave. As was the 
case in facing the Soviet Union, there may 
well be tensions today in choosing between 
serving SMO and SPO, on the one hand, 
and assuring adequate resources for WEI, 
on the other hand, as continued access to 
information needed for an understanding of 
enemy intentions and capabilities could be 
sacrificed by meeting the needs for immedi- 
ately actionable intelligence. 

Although warning and estimative intelli- 
gence may be seen as the core missions of 
strategic intelligence, they are also less tied 
to the details of ongoing operations in which 
the formal relationships between policymak- 
ers and intelligence provide a unique advan- 
tage and leverage for intelligence insights. 
Today's decisionmakers have many more 
sources of information than did their prede- 
cessors when the Intelligence Community 
was created; in turn, the Intelligence Com- 
munity holds far less of a monopoly over 
information about foreign events and tech- 
nology developments. Moreover, as one 
senior intelligence official noted, policymak- 
ers see themselves and their staffs as sub- 
stantively knowledgeable on issues of 
interest as the Intelligence Community and 
capable of serving as their own intelligence 
analysts. 28 As a result, users increasingly 
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26 Perhaps the scarcest resource is a senior decisionmaker's attention, which can easily be wasted. 

27 As an indication of the long time horizon involved in this function, both civilian and military defense officials need to 
look well into the future to develop strategy, plan forces, support research and development, and acquire systems. 

28 See Charles E. Allen, "Intelligence: Cult, Craft, or Business?" in "Comments of the Associate Director of Central 
Intelligence for Collection" at a Public Seminar on Intelligence at Harvard University, spring 2000. See http://pirp. 
harvard. edu/pdf-blurb.asp?id+51 8: 15. Henry Kissinger may be the most obvious example of this tendency, but it has 
continued since the Nixon administration and has come to include a far greater proportion of policy officials, especially 
as the sources of information on foreign developments have expanded dramatically and become available in near 
real-time. See Henry A. Kissinger, "America's Assignment," Newsweek, November 8, 2004: 38-43. 
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see themselves as participants in a process 
of judgments. 29 An experienced national- 
level user wrote recently, 

Today, the analyst no longer sets the 
pace of the information flow. The 
sources of information now available 
to the policy-level consumer... are far, 
far greater than a quarter century ago. 
It is almost a given that today's policy- 
level consumer of intelligence is well 
informed in his or her area of interest 
and not dependent on an intelligence 
analyst for a continuing stream of rou- 
tine, updating information. 30 



Implications of Differentiating Roles 

Careful differentiation among the three intel- 
ligence roles discloses dimensions that are 
both analytically important and more mean- 
ingful for contemplating intelligence reform 
than the usually misleading bimodal distinc- 
tions between national vs. military users or 
tactical vs. strategic objectives. What truly 
distinguishes these intelligence roles is their 
perspective and emphasis — a significant 
distinction that has been lost in recent argu- 
ments over intelligence reform. 

To begin with a particularly important point, 
a tactical or a strategic focus does not nec- 
essarily distinguish military from civilian 
users. 31 Moreover, the less quantifiable and, 
therefore, so/terinformation and analysis on 



individuals, decisionmaking, and social 
dynamics that used to be produced primarily 
for national users is now increasingly 
demanded to support military operations at 
the tactical level. Such information is inher- 
ently more judgmental and inferential — and, 
therefore, less precise — than analysis of 
physical or technical characteristics in 
orders-of-battle (OOBs) and tables of orga- 
nization and equipment (TOEs). It is less 
amenable to counting or to the gathering of 
external physical signatures by technical 
collection systems; it is more dependent on 
language skills, deep expertise on the 
region and cultures, and knowledge of the 
personalities. 32 It is also harder to validate 
or prove than estimates of technical factors. 
Such capabilities go beyond "reporting" that 
used to be the core of current intelligence. 

However, both SMO and SPO are, by nature, 
mission- or task-oriented and tightly focused 
on the problem at hand; and this narrowed 
focus has significant time and perceptual 
implications for analysts and the intelligence 
sources supporting them. 33 Given the stress, 
time pressures, and immediate — as opposed 
to potential — stakes attendant on current 
operations, human decisionmakers try to 
concentrate only on the immediate situation 
and the information relevant to it, while 
actively screening out other inputs. This is 
the intelligence analogue of human "foveal 
vision," which offers the highest visual reso- 
lution but also a very narrow field-of-view. 34 



29 As one senior intelligence official remarked in a private meeting, "We are all in the business of making judgments; 
but too many in the Intelligence Community continue to believe that they are instead providing crystalline analyses." 

30 Clift, "Intelligence in the Internet Era," Studies in Intelligence 47 , no. 3 (2003). 

31 This distinction might have been clearer before civilian leaders began taking detailed interest in overseeing tactical 
operations— as began during the Vietnam War with presidential interest in the selection of bombing targets. With 
respect to the Soviet Union during the Cold War, national and military users had clear areas of primary interest; 
national users were focused on intelligence illuminating key political, economic, technological, and social factors 
affecting national power and intentions, while military users were more focused on the likely force capabilities and 
doctrines of potential adversaries. After the Cold War, improving technical capabilities and the emergence of the 
"strategic corporal" combined to increase the interest of civilian policymakers in overseeing tactical operations. 

32 Several senior military participants at the Charlottesville conference highlighted these demands. See Charlottesville 
Conference Report, 2-5. 

33 There is a very large body of literature on the physical and psycho-perceptual effects on human judgment and 
decisionmaking under stress that is relevant to these distinctions. 
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In contrast, warning and estimative intelli- 
gence are the analogues of human "periph- 
eral vision," in which there is low resolution 
but a wide field-of-view. Peripheral vision is 
very sensitive to cues of dynamic change, 
which trigger anticipatory responses. 
Although warning is concerned with activat- 
ing the response cycle, and estimative intelli- 
gence is intended to create a frame of 
reference for the decisionmaker, both are 
intended, through preconditioning and antici- 
patory consideration, to enable a more 
appropriately and contextually sensitized 
response on the part of users. 35 

Another important implication of the differ- 
ing emphasis on decisions with a long-term 
view and those requiring prompt action — 
the classic distinction between strategic and 
tactical — concerns the nature of the advan- 
tage to be gained from the information, and, 
therefore, how it is exploited. In recent 
years, the tasks of intelligence, and its suc- 
cesses and failures, have focused on pro- 
viding immediately actionable (in this sense, 
tactical) intelligence to users — information 
that can provide a rapid or near-instanta- 
neous advantage, whether for interdicting 
hostile military forces, preventing terrorist 
incidents, or supporting diplomatic initia- 
tives. Emphasizing current intelligence for 
actionable exploitation may have created an 
unintended mind-set that undervalues the 
immense importance of knowing and under- 
standing the adversary's intentions through- 
out the course of the confrontation, even at 



the cost of foregoing exploitation of these 
sources for temporary advantage on the 
battlefield or in the diplomatic conference 
room. 36 This stress on current intelligence 
also influences the priorities among the 
types and attributes of information we col- 
lect, the nature of the collection and pro- 
cessing systems, the analytical methods we 
use, the stresses we place on analysts, and 
the metrics by which we assess the perfor- 
mance of intelligence. 37 

There is yet another important distinction 
between these roles. By looking out to the 
future, WEI is basically a surprise-prevent- 
ing function intended to heighten a policy- 
maker's ability to visualize the 
consequences of anticipated and unantici- 
pated events and to prepare for them men- 
tally; it is not designed to be "evidence- 
based truth-telling" that will stand up in 
court. In addition, as we better appreciate 
the implications of emergence and the 
emergent behaviors of complex adaptive 
systems, we need to place greater empha- 
sis on anticipation while recognizing that 
precise prediction or forecasting is even 
harder than previously understood. Appreci- 
ating the differences in perspective created 
by these roles is very important because 
failing to make clear distinctions between 
them may aggravate a major problem 
before the Intelligence Community: the dis- 
connect between the emphasis on current 
reporting or providing situational aware- 
ness, which must be evidence-based, and 



34 The fovea, a small pit at the back of the retina, forms the point of sharpest vision. The intensely narrow concentration 
of foveal vision is recognized as being a major contributor to "change blindness." 

35 One type of warning function amenable to focused monitoring involves potential surprise from a recognized 
adversary undertaking a feared but pre-identified activity (such as a Warsaw Pact invasion across the Inner-German 
border). This other warning function serves to guard against truly unexpected or unforeseen events. In both cases, 
they are designed to encourage thinking about, and contingency planning for, "surprises" before they occur. 

36 The widely repeated— but apocryphal— story that perhaps best exemplifies this understanding of "strategic 
intelligence" is that of Churchill's allowing Coventry to be bombed in order to safeguard the long-term informational 
advantages gained from Allied code-breaking achievements against the Axis. The immeasurable importance of such 
intelligence in the successful Allied efforts to interdict Rommel's supply lines during the North African campaign and 
in winning the crucial Battle of the Atlantic testify to these other equities with possibly higher priority. 

37 Barger, 26. Although her specific comment refers to the impact of precision (timeliness and resolution, for example) 
on the quality and quantity of intelligence, her larger point is that functional needs stemming from roles and missions 
drive what the Intelligence Community provides its users and how it does so. 
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the policymaker's need for anticipatory judg- 
ments, which, by nature, trade the confi- 
dence derived from hard evidence for an 
intuitive, or gestalt, understanding of the 
whole derived from inference. 38 It is unlikely 
that analysts will have firm evidence to offer 
the policymaker to support alternative inter- 
pretations of the future, and they will need to 
rely on inference and informed speculation 
that is persuasive to decisionmakers. 39 In 
particular, as one experienced intelligence 
analyst noted, 

"Getting inside the other guy's head" 
can only be conjectural because, in 
most cases, even the "other guy" 
doesn't know exactly why he's doing 
what he's doing. 40 

Even if there are predictive judgments to be 
made in both SMO and SPO, 41 they tend to 
have short time horizons and reasonably 
short inferential chains; as the predictive 
time-constants are short, observation of 
adversary actions can serve to validate or 
disprove these judgments and thereby 
improve confidence in them and the ana- 
lyst's judgment. 

Those providing SPO, in particular, must 
continually walk a fine line between serving 
the policymakers' needs for relevant, 
focused, direct support and maintaining 
objectivity in providing the evidence and 
analysis. Staying close to the evidence 
assists the analyst in walking this line. At the 
same time, the author of this monograph 
noted a clear consensus among senior intel- 
ligence officers at a recent non-attribution 
conference that analysts can best serve pol- 



icymakers by offering them thoughtful and 
thought-provoking views that challenge 
their assumptions. It must be recognized, 
however, that helping to alter policymakers' 
assumptions is intruding directly into the 
policymaking process and, thereby, cross- 
ing the boundary that Sherman Kent tried to 
establish. As the policymakers demand 
judgments on actions and consequences 
farther in the future (moving the intelligence 
role from SPO to WEI), not only will the 
intrinsic uncertainties increase, but also the 
potential for tensions between policymaker 
and analyst over the objectivity (and validity) 
of the judgments and the conflicts among 
differing judgments. 42 

Another of these distinctions affects intelli- 
gence requirements and planning. Unlike 
SMO and SPO, where the users can clearly 
identify their areas of interest, priority 
issues, and information needs, the Intelli- 
gence Community must look beyond its 
users' perceptual horizons if it is to perform 
warning and estimative functions effec- 
tively. Almost by definition, with anticipatory 
intelligence, policymakers will be unable to 
tell the community where to look. Unfortu- 
nately, although the Intelligence Community 
must recognize that attempting to divine 
requirements for warning and other antici- 
patory intelligence from the users is not 
likely to be fruitful, it also must appreciate 
that it alone will bear the blame for failing to 
warn against the inevitable surprises arising 
from outside the fields-of-view of users. This 
demands, in turn, that the Intelligence Com- 
munity have some discretion and flexibility 
to allocate resources in areas not currently 
considered to be priority targets: listening 



38 Gestalt, a German word meaning "form" or "shape" is used in psychology to connote holistic understanding of the 
entirety of a phenomenon. This follows Kendall's approach of "creating pictures," as noted by Jack Davis in "The Kent- 
Kendall Debate of 1949," Studies in Intelligence 35, no. 2 (1991). 

39 This is a comment to the author by a senior intelligence officer who has served in both roles. 

40 Private communication to the author from John Bodnar, 3 November 2004. 

41 The military, in particular, is increasingly emphasizing "predictive battlespace intelligence" as a central component 
of "information superiority." It is usually, however, a different kind of prediction than that required to support SPO. 

42 As the founder of Air Force and Joint Staff Studies and Analyses, Lt. Gen. Glenn Kent (USAF, ret.), paraphrasing 
Shakespeare, once warned analysts, "Neither a prostitute nor a proselytizer be." 
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too closely to the customers, and looking 
only where directed, guarantees future stra- 
tegic warning failures. 

It is absolutely essential that the Intelligence 
Community and those who depend on it 
understand the principal distinctions 
between these two functions. In their con- 
clusions about the nature of the Intelligence 
Community's problems, the extraordinary 
differences between the report of the 9/11 



Commission and that of the SSCI on Iraqi 
WMD reveal the dangers of conflating the 
two distinct functions or ignoring the differ- 
ences between the three roles. When one 
tries to assess the adequacy of Intelligence 
Community performance across these 
domains, identify shortfalls, or prescribe 
changes — whether in business practices, 
tools, or organizational arrangements — the 
appropriate answers will almost certainly 
differ greatly from one role to the other. 
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As astute members of the Intelligence Com- 
munity have observed, intelligence capabili- 
ties (and the organizations that provide 
them) are not general purpose tools but, 
rather, specialized and "finely honed" instru- 
ments that evolve and adapt to the specific 
challenges and requirements placed upon 
them. 1 Many of today's principal analytic 
problems arise from continued reliance on 
analytic tools, methodologies, and pro- 
cesses that were appropriate to the static 
and hierarchical nature of the Soviet threat 
during the Cold War and were, in that envi- 
ronment, largely successful. We possessed 
several decided advantages that enabled us 
to overcome some of the limitations on our 
capabilities: a careful and cautious "Main 
Enemy" that was also a far simpler and 
slower target than those we face today; 
more time; many and more experienced 
analysts; and varied and well-established 
sources of information, including the adver- 
sary's own vast outpouring of seemingly triv- 
ial but very useful data. 

Given these advantages, the Intelligence 
Community was able to: 

• Concentrate, focus, and build a deep 
foundation of cumulative evidence; 

• Foster longstanding and deep expertise in 
its analytic cadre by exploiting a very 
dense information environment; 

• Rely on multiple-source collection, which 
generally allowed us to cross-check 
information and make judgments 
consistent with the highest professional 
standards; 



• Largely neglect intelligence collection and 
analysis on "soft" cultural, societal, and 
people issues (other than the most 
prominent elites), because the plans and 
intent of the adversary were in the hands of 
a small Politburo and because we "knew" 
that knowledge of the plans and intent of 
subordinate elements or nations — for 
example, Poland— was rarely necessary; 2 
and 

• Employ an intelligence model that could 
rely on collecting "secrets" in voluminous 
quantities and required a mass production 
approach for producing reports. 3 

That was also a period of relative information 
scarcity on worldwide events, and the Intelli- 
gence Community had a substantial com- 
parative advantage over other information 
providers through access to intelligence 
obtained by clandestine collection. 

The United States had many Cold War suc- 
cesses, of course, but there were always 
significant shortcomings in American intelli- 
gence capabilities. These shortcomings 
resulted in surprises that had important and 
unanticipated consequences for US policy. 
Among these were the Egyptian/Syrian 
attacks on Israel opening the 1973 Yom Kip- 
pur War, the 1 974 Indian nuclear test, the fall 
of the Shah and the accompanying ascen- 
dance of a fundamentalist Islamic regime in 
Iran in 1 979, the unexpectedly rapid collapse 
of the Warsaw Pact in 1 989, and the disso- 
lution of the Soviet Union in 1991. 

The traditional intelligence methods were 
even less successful against some impor- 
tant targets that have carried over into the 
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1 See Aris A. Pappas and James M. Simon Jr., "The Intelligence Community: 2001-2015," Studies in Intelligence 46, 
no. 1 : 39-47; Bruce Berkowitz, "Intelligence for the Homeland," SAIS Review24, no. 1 (Winter-Spring 2004): 3. 

2 To appreciate the costs of this view, see the brief account on page 41 of Professor Murray Feshbach's use of Soviet 
and Russian health statistics to derive an important conclusion. 

3 The bureaucratized nature of the Soviet Union and the state of telecommunications throughout the Cold War allowed 
the United States to exploit its technological prowess and employ effective remote collection capabilities in addition to 
the traditional method of using human sources. 
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post-Cold War era — such as Iran, China, 
North Korea, nuclear proliferation, and the 
global WMD black-market. In these cases 
(exemplified by the 1998 Indian and Paki- 
stani nuclear tests), we have had serious 
shortcomings in understanding because, in 
the face of the numerous, continuing, and 
competing demands for higher priority cur- 
rent intelligence, we were unwilling or 
unable to make the long-term commitments 
to build the deep knowledge base the task 
required. 

Today, the United States may need to take 
action more frequently on the basis of 
ambiguous intelligence, against harder to 
recognize threats, from less risk-averse 
enemies — especially the shifting groups of 
radical Islamic fundamentalists who foment 
terrorist activities and may wield capabilities 
that could devastate cities, societies, or 
economies. These new Islamic adversaries 
also pose, as did the Soviet Union, a long- 
term ideological challenge whose character 
and "operational code" we do not currently 
understand. Unlike the Soviet Union, how- 
ever, which was a hierarchical, bureaucratic 
state whose organizational character we did 
understand, the new targets are more 
dynamic and complex. These new groups 
morph frequently and metastasize like can- 
cers, emerging with new personalities and 
network linkages and threatening new tar- 
gets. 4 

As a result, today's Intelligence Community 
must contend with: 

• Little foundational knowledge on major 
adversaries and cultures; 



• Fragmentary evidence and sparse 
intelligence flows, which often cannot be 
substantiated or contextualized — even 
against the highest priority current targets; 

• Thin domain ("account") expertise, due to 
multiple and shifting priorities and frequent 
shuffling of relatively inexperienced 
analysts; 5 and 

• An analysis model that remains heavily 
dependent on "secrets," even when the 
key intelligence questions may involve 
mostly "mysteries" and "obscurities." 6 

Aggravating these existing shortcomings, a 
cascade of significant and complex devel- 
opments that will pose substantial new chal- 
lenges is already evident, making the 
adequacy of the community's current capa- 
bilities even more problematic. 7 These 
changes involve more dynamic geostrategic 
conditions, more numerous areas and 
issues of concern, smaller and more agile 
adversaries, more focus on their intentions 
and plans and less on large physical objects 
and weapons systems, a more open infor- 
mation environment, and more widespread 
understanding of American intelligence 
methods and capabilities by our adversar- 
ies. Additionally, the sense of vulnerability to 
terrorist attack on US territory left by 9/11 
has created huge new demands on the 
Intelligence Community to provide informa- 
tion for homeland security. 8 The immensity 
of these challenges complicates the task of 
developing appropriate cures for the real 
causes of inherited shortcomings and of 
heading off new analytic shortcomings. 



4 Marc Sageman, "Understanding Terror Networks," FPRI E-Note, 1 November 2004. 

5 In addition, inexperienced analysts may lack the ability to tap sources of deep expertise available through 
connections to long-standing professional networks. 

6 Fritz Ermarth originally developed this typology. For the details, see page 40. 

7 Barger, 28. 

8 At the same time, the breaching of old distinctions between foreign and domestic intelligence activities has 
increased public concern over potential threats to privacy. 
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The lengthy reports of the 9/1 1 Commission 
and the SSCI report on Iraqi WMD laid out 
in great detail their views on why these two 
failures occurred. Although they occurred in 
very different substantive domains, at very 
different levels of analysis, in different parts 
of the intelligence organizations, and exhib- 
ited significantly different failure modes and 
causative reasons, both reports tended to 
locate the causes in problems related to 
information sharing and coordination, 
instances of insufficient collection and poor 
data, and "errors" of analytic judgment by 
individuals and groups. 9 

The Intelligence Community now finds itself 
under intense scrutiny and faced with the 
need to transform in fundamental ways in 
order to meet the entire range of national 
security intelligence challenges only par- 
tially recognized in the legislatively man- 
dated reforms. Addressing these 
challenges requires fundamentally new 
approaches in both collection and analysis 
as well as in the processing and dissemina- 
tion methods that support them. As Barger 
aptly notes: what is needed is revolution, not 
reform. 10 



Misunderstanding Analytic Processes 

The litany of failures should have been a tip- 
off to the deep-seated nature of the analytic 
problems. Such a series of "idiosyncratic" 
errors by individuals and small groups within 
an organization are, however, more likely to 
be symptoms than root causes, as Perrow 
convincingly demonstrated in case studies 
of Three Mile Island and Chernobyl. 11 A pat- 



tern of repeated errors is often a signal of 
seriously dysfunctional methods — funda- 
mental and systematic failures of proce- 
dures and processes throughout an 
organization. 12 From this perspective, the 
proximate causes of the failures identified in 
both the report of the 9/1 1 Commission and 
the SSCI report on Iraqi WMD hardly appear 
to be convincing root causes of these recent 
intelligence failures. 

A more accurate diagnosis of the sources of 
our intelligence shortcomings requires a 
deeper and more thoughtful analysis of why 
organizations make mistakes — causes that 
go beyond obvious superficial conditions 
created by flawed organizational structures 
and insufficient directive authorities. As 
Charles Sabel noted, "There he [Herbert 
Simon in Administrative Behavior] showed 
that modern organizations were efficient 
precisely because they systematically 
turned habits — the disposition to react to 
particular situations in a characteristic, but 
open-ended way — into rigid routines." 13 Not 
unexpectedly, these routines "work" for the 
specific conditions they were developed to 
address. They rarely perform well for off- 
design conditions, however, and, often, the 
better they work for the design conditions, 
the more narrow the set of conditions for 
which they are appropriate. Paradoxically, 
the better they work, and, therefore, the 
more efficient the organization at its routine 
tasks, the greater the danger that the orga- 
nization will fail to be sensitive to its environ- 
ment and changes occurring there. As with 
the dinosaurs, scores of major American 
corporations have fallen victim to this pat- 



9 Indeed, the IRTPA was largely driven by the recommendations of the 9/11 Commission, which focused mostly on 
one, albeit important, aspect of intelligence needs, that of counterterrorism. But, as noted above, the IRTPA reforms 
mandate changes that affect all functions of the Intelligence Community. 

10 Barger, 28. 

11 Perrow, Normal Accidents. 

12 For examples of the consistent nature of such errors, see the following: the "Jeremiah Report"; Interview with 
Richard Kerr, MSNBC, 14 July, 2003 (concerning the "Kerr Report" on the Iraqi WMD NIE). An unclassified portion of 
that report is in Studies in Intelligence 49, no. 3 (2005) (hereinafter cited as Kerr, et al.). 

13 Charles F. Sabel, "Theory of a Real Time Revolution." 
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tern of "over adaptation" and "change blind- 
ness." 14 The Intelligence Community runs 
the same risk. 



The Problem of the Wrong Puzzle 

Frequent public references to "failing to con- 
nect the dots" are especially problematic for 
an accurate understanding of intelligence 
errors and failures. This view of the analytic 
shortfalls is particularly perverse, because it 
masks the true nature of the analyst's chal- 
lenges. The flawed "connect the dots" anal- 
ogy flows from the image of the children's 
game book in which lines are to be drawn 
between a set of numbered dots in order to 
make a recognizable picture. That analogy 
assumes, however, that — as in the chil- 
dren's book — the dots exist, that it will be 
obvious which dots connect to which others 
and in what order. 15 The problem is that this 
simple analogy overlooks a well-known 
phenomenon in psychology that is often 
illustrated by the "Rubin Vase Illusion": that 
evidence really does not "speak for itself"; 
rather, that information is "perceived and 
interpreted." 16 Humans are extremely good 
at finding patterns, even when there is 
none — hence the classic intelligence apho- 
rism, "You rarely find what you're not looking 
for and usually do find what you are looking 
for." 17 

If we are to use a puzzle analogy, perhaps a 
more appropriate model might be that of a 
guest at a resort hotel who, on a rainy after- 
noon, wanders into the game room and 



finds a box holding a large number of jigsaw 
puzzle pieces. As the cover of the box is 
missing, there is no picture to guide him in 
reconstructing the puzzle, nor is there any 
assurance that all the pieces are there. 
Indeed, when he discovers that there are 
several other empty puzzle boxes on a 
shelf, it is not even clear that all the pieces 
in the box belong to the same puzzle. 
Reconstructing the puzzle in this example is 
a far different and more difficult challenge 
than linking numbered dots, where the out- 
line of the image is reasonably apparent. 18 

Both the dots analogy and the model of evi- 
dence-based analysis (discussed in the fol- 
lowing section) understate significantly the 
need for imagination and curiosity on the 
part of the analyst. 



The Myth of "Scientific Methodology" 

Many well-informed outside commentators 
and intelligence professionals continue to 
talk about the "science of analysis," and only 
some of them are truly aware of the shaky 
foundations of this belief or of its real impli- 
cations. 19 But this talk of a "science of anal- 
ysis" is a conceit, partly engendered by 
Sherman Kent's dominating view of intelli- 
gence analysis as a counterpart of the sci- 
entific method. 20 The reality is otherwise; 
analysis falls far short of being a "scientific 
method" in the common, but usually misun- 
derstood, sense. Moreover, this view of sci- 
ence itself is "scientism," which fails to 
recognize the important role of less "ratio- 



14 See Carol Loomis, "Dinosaurs?" Fortune, 3 May 1993: 36ff. An accompanying sidebar recounted how a senior 
Sears executive pointed behind himself to the tens of volumes of corporate practices and rules that governed the 
corporate response to any conceivable problem. The emergence of mid-market national discount chains wasn't 
covered; and, therefore, "...wasn't a problem they had to address." 

15 This relatively simple problem is known formally in mathematics as a "directed graph." 

16 Edgar Rubin, 1915. Heuer discusses such perceptual problems using different examples in Chapter 2 of The 
Psychology of Intelligence Analysis. 

17 This remark is often attributed to Amrom Katz, a pioneer in aerial and overhead reconnaissance. 

18 Heuer, Chapter 6. 

19 See, for example, Frank Hughes and David Schum, Evidence Marshalling and Argument Construction. 

20 Sherman Kent, Strategic Intelligence. See also Jack Davis, "The Kent-Kendall Debate." 
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rial" and less "scientific" elements, such as 
imagination and intuition. 21 As Mark and 
Barbara Stefik, knowledgeable and 
respected participants in the discipline of 
science, have written about science and 
innovation in a recent book. 

The word "theory" usually connotes a 
formal way of thinking logically or 
mathematically. In this formal sense, 
theory takes its place in a knowledge- 
generating process called the scientific 
method. The scientific method 
includes hypothesis formation, experi- 
ment planning, execution, and data 
analysis. In this scheme, theory is 
used to make predictions. Theory is 
created by a process that includes 
hypothesis formation and testing. 

Unfortunately, this notion of theory and 
the working methods of science and 
invention leaves out imagination. This 
makes it both boring and mislead- 
ing.... 22 

Citing a well-known commentary by a Nobel 
laureate, the Stefiks add: 

In [Peter] Medawar's view, the stan- 
dard scientific paper promotes an error 
of understanding how science is done 
because it confuses proving with dis- 
covering. 23 The layout of a scientific 
paper makes it appear that the doing of 
science is the careful laying out of 
facts. Once the facts are in, the conclu- 
sions follow. This makes it seem like 
science is all about deduction. Unfortu- 



nately, this formal structure leaves out 
the creative part of discovery and 
invention. The structure of a scientific 
paper is only about proof, promoting 
the systematic marshalling of evi- 
dence. In this abbreviated story, once 
a scientist has by some means figured 
it out, the paper lays out the conclu- 
sions logically. 24 

A more realistic and useful appraisal of the 
process of intelligence analysis comes from 
Charles Allen, a long-time senior intelli- 
gence official: "I want to speak mainly about 
the art and craft of intelligence.... We could 
have talked about the science of intelli- 
gence, but, by and large, as far as I'm con- 
cerned, the science of intelligence is yet to 
be invented. I don't see it. It's not really 
there." 25 This is not to suggest that rigor, 
accuracy, clarity, and precision are not 
required in intelligence analysis; given the 
stakes, they are obviously essential. But 
demanding a false precision from an analy- 
sis process that is itself incorrectly modeled 
on a common misunderstanding of the 
methods of science is not likely to improve 
the quality of analysis. Indeed, an important 
issue for both managers and users of anal- 
ysis to consider is the likelihood that there 
may be little concordance between preci- 
sion in the details of the answer and the 
accuracy of the overall (gestalt) judgment. A 
process and methodology too focused on 
provable evidence may get the details right 
at the cost of ignoring important inferential 
judgments that need to be conveyed in 
order to provide a true sense of the uncer- 
tainties of both evidence and judgment. 26 



A process and 
methodology too 
focused on prov- 
able evidence 
may get the 
details right at the 
cost of ignoring 
important infer- 
ential judgments. 



21 The term "scientism" is used to connote the frequent confusion between the appearance of a formal scientific 
methodology and the actual conduct of science, which may be intuitive, but is nonetheless subject to rigorous proof. 

22 Mark Stefik and Barbara Stefik, Breakthrough: Stories and Strategies of Radical Innovation, 110. 

23 The Stefiks are referring to Peter Medawar's article, "Is the Scientific Paper Fraudulent? Yes; It Misrepresents 
Scientific Thought," published in the Saturday Review 47 , 1 August 1964: 42-43. 

24 Stefik and Stefik, 110-11. 

25 Allen. 

26 For more on this subject, see the section on "Evidence-based Scientism" in Chapter Three. 



27 



Chapter Two: Assessing Critical Analytical Shortfalls 



The Flaws of a "Tradecraft" Culture 

Intelligence analysis remains largely a craft 
culture that is conducted within a self-pro- 
tective guild system and taught by means of 
a broken apprenticeship process. There are 
other fields, such as science, medicine, and 
warfare, in which knowledge is also under- 
stood to be tentative and not subject to for- 
mal "proof," as is possible in mathematics. 
Within such professions, the cumulative 
practices, habits, and mindsets of an 
evolved culture are especially important for 
the creation of knowledge and the transmis- 
sion of expertise. As with intelligence, these 
other communities are ones in which much 
of the knowledge needed for effective per- 
formance relates to the often-arcane pro- 
cesses of the craft (tradecraft, as the 



Intelligence Community terms it). This 
knowledge is tacit and difficult to elicit from 
the experts, and it is usually best communi- 
cated by personal example and practice. 27 
However, the culture of intelligence lacks 
many of the formalized processes, such as 
"peer review," and the cumulative knowl- 
edge structures that the academic, military, 
and medical communities have created to 
address similar challenges in building a 
solid foundation of understanding that can 
be passed to successor practitioners. Per- 
haps for these reasons, intelligence analy- 
sis is not yet a true profession. Within this 
culture, therefore, effective mentorship is 
especially important for transmitting exper- 
tise and, perhaps more significantly, for 
imparting professional standards and val- 
ues to apprentices. 
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27 The professions of medicine and law refer to themselves as "practices," which reflects their roots in the guild 
system. More importantly, perhaps, this usage conveys that the essential elements of a profession (ethos, ethics, and 
skills) are human values best transmitted by people. 
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Although many of the reform proposals 
responding to the recent intelligence failures 
took diametrically opposed positions with 
respect to the role of the DCI, the creation of 
the DNI, and the functions of the CIA within 
a "transformed" Intelligence Community, 
they were mostly quite similar in that they 
focused both on wiring diagrams and formal 
authorities as the mechanisms for reform 
and on creating a "czar" to execute those 
authorities through centralized manage- 
ment. 1 As is often the case, reform legisla- 
tion took a structural route to address 
problems that flow from dysfunctional pro- 
cesses, inbred cultural practices and habits, 
and failures of human leadership. 

Although the legislative changes have 
restructured the community, other than 
emphasizing "competitive analysis" and 
more HUMINT, the IRTPA contains little sig- 
nificant language about reforming the inter- 
nal functional processes of intelligence- 
collection and analysis. The legislation failed 
to take into account several key distinctions 
that any intelligence reform proposals must 
recognize: differences in issue domains; tac- 
tical vs. strategic consequences; narrowly 
focused vs. synoptic collection and analysis; 
reporting vs. deep analysis; immediate vs. 
long-term decision horizons; foveal vs. 
peripheral perception; action vs. exploita- 
tion; and defined, tasked requirements vs. 
speculative information gathering. 

Some recommendations, in fact, could 
aggravate the already dysfunctional condi- 
tions, further damaging analytic capabilities. 
The IRTPA directed the establishment of 
centers for priority issues, as recommended 
by the 9/11 Report, but these are likely to 
place even more emphasis on current intelli- 



gence, whether or not this was intended. 
Integrated centers combining operators and 
analysts do foster better sharing and collab- 
oration among them, but they invariably 
place the emphasis for analysts on opera- 
tional support. Moreover, creating static 
structures and rigid processes designed to 
deal with fixed areas or issues is not a par- 
ticularly sound response to events and con- 
ditions that are very fluid and fuzzy. These 
can be better handled by operating within a 
process that allows flexibility and discretion. 
Finally, with an already severe shortage of 
effective managers and experienced intelli- 
gence officers (including analysts, opera- 
tors, and collectors), establishing more 
centers with mandated institutional positions 
and priorities can only further dilute an 
already over-stressed cadre of intelligence 
professionals. 



An Inventory of Analytic Pathologies 

Curing analytic shortcomings cannot be 
done by making minor modifications to the 
existing processes or even by wholesale 
replacement or upgrading of the analytic 
cadre. 2 Although it may be tempting to focus 
on better educating and training the ana- 
lysts — the better for them to be able to "con- 
nect the dots" — altering the analytic model 
and the processes on which that model 
relies is almost certainly a more appropriate 
response to the profound problems beset- 
ting intelligence analysis. Without fixing the 
fundamental shortcomings in analytical pro- 
cesses, the system will always depend on 
the ability of individuals to work around the 
impediments, which adds additional stress 
to an already burdensome set of tasks. 



1 See, for example, a bill proposed by Rep. Jane Harman, the "Intelligence Transformation Act of 2004" (HR41 04): "The 
goal. ..should be to enhance the DNI's ability to coordinate and integrate operations, focus the community on priorities, 
share information better. ..and so on, but not detract from the support that cabinet secretaries need and expect from 
their intelligence organization, and not dilute competitive analysis." 

2 One excuse for the very heavy cuts in mid- and senior-level analytic cadres in the early 1 990s was that it was 
necessary to weed out those whose expertise was presumed to be passe and, therefore, no longer needed. 



. . . establishing 
more centers 
with mandated 
institutional po- 
sitions and prior- 
ities can only 
further dilute an 
already over- 
stressed cadre 
of intelligence 
professionals. 
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At the same time, it should be acknowl- 
edged that there are limitations on the depth 
and quality of the expertise available within 
the community. The past decade has shown 
that the Intelligence Community is likely to 
find itself behind academic and commercial 
sectors in many frontier areas simply 
because, for the most part, that is where 
these frontiers are first explored. Further- 
more, given the unpredictable nature of 
emerging challenges and the often short 
response time they allow, the community is 
unlikely to have on hand sufficient regional, 
cultural, linguistic, technical, or specialist 
expertise to meet high priority threats to our 
national interests. Therefore, unless com- 
munity management decides to forgo hav- 
ing all needed expertise within the 
community — to open its boundaries, 
remove the barriers, and create mecha- 
nisms to draw on external expertise and 
knowledge — these impediments to exploit- 
ing outside expertise will frustrate its ability 
to meet emerging mission challenges. 

Interestingly, even if the old analytic meth- 
ods and processes developed during the 
Cold War were themselves capable of 
addressing the emerging security chal- 
lenges of post-Soviet adversaries, a review 
of postmortems of intelligence failures prior 
to 9/11 and the Iraqi WMD fiasco showed 
that these processes and procedures were 
routinely violated, frequently leading to fail- 
ures. 3 In addition to shortcomings in the 
basic analytic paradigm and processes, 
according to the study, the system lacked 
enforceable self-correcting features and 



functioning compliance mechanisms. This 
suggests that there has been a more funda- 
mental failure in leadership and in the basic 
institutional and management mechanisms 
for assuring effective performance of over- 
sight, assessment, accountability, and 
responsibility. That these failures persisted 
for so long should be of concern to mem- 
bers of the analytic community, their leader- 
ship and management, and the oversight 
bodies as well. It is a warning sign that bet- 
ter oversight and enforceable compliance 
procedures are needed, because the ana- 
lytic community itself may no longer pos- 
sess the internal self-discipline or 
professional standards to do so. 4 This also 
clearly implies that greater attention to pro- 
fessional ethos and standards must be an 
integral part of efforts to transform analysis. 

Indeed, well before 9/11 , several articles 
written by experienced community officials 
pinpointed fundamental shortcomings 
within the community's analytical capabili- 
ties and highlighted dysfunctional pro- 
cesses as the causes. 5 Thus, the 
longstanding failure modes within the exist- 
ing intelligence analysis paradigm must be 
identified and corrected, along with the 
management and oversight procedures, if 
the community is to meet new needs. 6 

Two elements of the current paradigm are 
especially worthy of attention. First is the 
inefficiency of the "account" structure. 

The account system, by its very nature, cre- 
ates institutional and individual "ownership" 
of important intelligence domains. The ben- 



3 See Jack Davis, "Improving CIA Analytic Performance: Strategic Warning," and "Improving CIA Analytic 
Performance: Dl Analytic Priorities." 

4 Several retired, formerly very senior, CIA officials have made this point about erosion of professional elan— ethos, 
ethics, and standards— that help keep the analyst from, as General Kent put it, succumbing to the temptations of 
prostitution or proselytizing (see footnote 43 on page 22). One attribute of a profession, especially a "learned 
profession," is that members can work without supervision. This is also one of the characteristics that define 
journeymen in a craft system. 

5 Carmen Medina, "What To Do When Traditional Models Fail," Studies in Intelligence 46, no. 3 and Russ Travers, 
"The Coming Intelligence Failure," Studies in Intelligences, no. 2. 

6 See Berkowitz, 3. That each organization develops tradecraft and evolves practices that may not transfer easily if 
challenged by new functions complicates the community's ability to shift its focus and assets to meet new challenges. 
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efit is that it provides a basis for accountabil- 
ity; the disadvantage is that ownership 
inhibits sharing, cooperation, and collabora- 
tion. It has also encouraged "stovepiping" by 
collection discipline and the control of infor- 
mation by collectors through originator-con- 
trol (ORCON) restrictions. A fundamental 
redesign of analysis should start by disman- 
tling both the notion of information owner- 
ship and the paradigm of "accounts," while 
maintaining accountability for performance; 
information must become the common 
property of the community, and someone 
with authority must oversee sharing and 
configuration management. 7 Such changes 
are likely to involve altering the existing 
institutional mechanisms in order to forge 
interagency virtual "clusters" to perform the 
analytic functions while creating "mission 
managers" with the authority to assure that 
important subjects and user needs are prop- 
erly serviced. 

The second element is the Intelligence 
Community's strong cultural orientation 
towards an "evidence-based scientism." 

Although this approach may be appropriate 
for the current intelligence functions that rely 
heavily on gisting and reporting and domi- 
nate both SPO and SMO, it clearly limits the 
ability of analysts to address the anticipatory 
intelligence needs of decisionmakers, which 
usually demand more reliance on judg- 
ments and inference chains and less on 
specific evidence. In return for more focus 
on the WEI role, decisionmakers who rely 
upon anticipatory intelligence must realize 
its inherent properties and limitations and be 



prepared to accept greater uncertainty in 
assessments and estimates in order to 
obtain better gestalt understanding. 8 

We need to understand that "warning" is 
largely built on modeling (either explicit or 
implicit) and synthesis, which are deductive 
processes, and not on analysis, which is an 
inductive process. In all cases, analysis pro- 
cesses should also access more non-tradi- 
tional sources and incorporate a wider 
range of information to construct corrobora- 
tive fabrics that can confirm or disconfirm 
critical information and hypotheses. The 
rigor and strength of these methods must 
rely less on narrow tests of the quality of the 
evidence or adherence to formalisms than 
on analysts being sufficiently "mindful" to 
recognize the pitfalls they may encounter in 
blind application of approved methodolo- 
gies. 9 In addition, there is a huge research 
literature on decisionmaking under uncer- 
tainty that could be exploited to introduce 
innovative techniques for analysis and deci- 
sion support. What all this comes down to, 
however, is not to disregard the need for 
rigor of method and quality of evidence, but, 
rather, to suggest that a construct of analy- 
sis too narrowly tied to a misunderstood 
"scientific method" needs to be augmented 
and leavened with intuition, curiosity, and a 
thirst for discovery— all essential elements 
of good science. 10 

Beyond these two fundamental changes, 
eight other problematic features of the cur- 
rent intelligence environment need to be 
addressed. 



A fundamental re- 
design of analy- 
sis should start 
by dismantling 
both the notion of 
information own- 
ership and the 
paradigm of "ac- 
counts. " 



7 Creation of an "information commons" must be done in full awareness of the problems associated with the 
conundrum of commons, the conflict for resources between individual interests and the common good. Garrett 
Hardin, "The Tragedy of the Commons," Science 162 (1968): 1243-8. 

8 It is like the saying, "better to be approximately correct than precisely wrong." Josh Kerbel, "Thinking Straight: 
Cognitive Bias in the US Debate About China," Studies in Intelligence 48, no. 3 (2004). 

9 In this regard, several commentators, among them Sabel, Fishbein, and Treverton, have recognized the potential 
applicability of practices drawn from "High Reliability Organizations" (HROs). Both this issue and "mindfulness" will 
be addressed in greater depth in Chapter Four. 

10 This thought owes much to comments by Stephen Marrin in a private communication on 1 November 2004. 
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3. The Tyranny of Current Intelligence 

Over the past decade, the Intelligence Com- 
munity's efforts to be responsive to its cus- 
tomers' demands for current intelligence 
have dominated collection and analysis. As 
a senior analyst noted, "The Intelligence 
Community really [is] focused on current 
intelligence, on policy support. It does very 
little research. It has very little understand- 
ing below the level of the policymaker and, 
in my view, on many issues. I think that, in 
some ways, these two groups are reinforc- 
ing each other's worst habits." 11 The disap- 
pearance of the long-term Soviet threat and 
the chaos of the post-bipolar geostrategic 
environment shifted policymaker interests 
to a series of crises du jour. Policymakers 
demanded that intelligence for their current 
problems receive priority as the US and 
Allied interventions in Somalia, Haiti, Bos- 
nia, Kosovo, and elsewhere proceeded. At 
the same time, the Pentagon's increasing 
demands for support in ongoing military 
operations, including such prolonged activi- 
ties as Northern and Southern Watch in 
Iraq, placed tremendous demands on the 
intelligence system for a continuing stream 
of timely support products. 12 

Prior to the 1 991 Gulf war, no commanders 
of regional or functional military forces — 
other than those for strategic forces — could 
expect to receive direct and timely support 
from national assets, especially to support 
tactical operations. 13 That war, and the 
glimpse it provided of the power of modern 



information systems to overcome shortcom- 
ings in C4ISR (command, control, commu- 
nications, computers, intelligence, 
surveillance, and reconnaissance) fueled 
an intense push to obtain "information supe- 
riority." This, in turn, created nearly insatia- 
ble appetites among military commanders 
for real-time intelligence support and gave 
birth to the concept of "net-centric warfare" 
(NCW). 14 Many of the new C4ISR systems 
(including national systems), and much of 
the effort and funds expended by the Intelli- 
gence Community since the Gulf War, have 
focused on providing direct, real-time sup- 
port to forces engaged in combat by closing 
the "sensor-to-shooter" loop and to meeting 
the information needs of the senior-level 
commanders directing those operations. 
When there are American forces deployed 
in active military operations, as there have 
been on a near-continual basis since the 
end of the Cold War, the highest priority is 
now accorded to providing intelligence to 
support them. 



4. Overemphasis on Production 

At the same time as demands to support 
military operations generated huge require- 
ments across the Intelligence Community, 
standing collection and analysis require- 
ments to fill databases and to produce rou- 
tine scheduled products set priorities for a 
substantial part of the intelligence "phase- 
space." The entire intelligence system is 
dominated by the demands of processing 



11 Intelligence and Policy: The Evolving Relationship, Center for the Study of Intelligence, June 2004, 7. 

12 These are not new problems. Defense Secretary Donald Rumsfeld, as chairman of the Commission on Ballistic 
Missile Threats, highlighted the failure of executive and legislative leadership to establish appropriate priorities while 
besieging the community with ad hoc taskings. See Side Letter to the Rumsfeld Report, 1 8 March 1 999, 2. 

13 Those needs and the impediments on access to national systems underwrote the very large military programs 
known as TIARA (Tactical Intelligence and Related Activities). National systems are also known as National 
Technical Means, the term used to refer to them in arms control agreements. 

14 In Network Centric Warfare: Developing and Leveraging Information Superiority David S. Alberts, John J. Garstka, 
and Frederick P. Stein define NCW as "an information superiority-enabled concept of operations that generates 
increased combat power by networking sensors, decisionmakers, and shooters to achieve shared awareness, 
increased speed of command, higher tempo of operations, greater lethality, increased survivability, and a degree of 
self-synchronization. In essence, NCW translates information superiority into combat power by effectively linking 
knowledgeable entities in the battlespace." 
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huge amounts of information gathered by 
collection systems whose architecture was 
largely designed during the Cold War to 
address a very different problem. This huge 
inflow created a production-oriented model 
and an "efficiency paradigm" better suited to 
the "Industrial Age" than to the Information 
Age of the Twenty-first century. 15 The vol- 
ume of collected intelligence is so vast that, 
even with automated assistance, human 
analysts can effectively review and evaluate 
only a small part of the flow. 

The existing paradigm for intelligence anal- 
ysis and dissemination still relies largely on 
published paper reports as the mechanism 
for delivering products to users. Further- 
more, without effective metrics to assess 
the value of intelligence to decisionmakers 
as well as the impact of analysis on the 
quality of their decisions, it is simple mea- 
sures of data collected, traffic processed, 
and reports produced that have influenced 
critical decisions on priorities and resource 
allocation within the Intelligence Commu- 
nity. Meeting the new challenges will require 
a greater emphasis on adaptability and agil- 
ity, as well as processes that are better able 
to respond to non-routine requests and 
high-priority challenges. The price to be 
paid for this, however, may be a reduction in 
efficiency as measured by the usual output- 
oriented metrics of mass-produced prod- 
ucts normally used in cost-benefit calcula- 
tions of routinized processes. Such 
calculations underweight the ability to meet 
unexpected situations and the "slack" that is 
usually essential in an adaptable organiza- 
tion. 



5. Over-Reliance on Previous Judgments 

The problem of "finished intelligence" stems 
from the conceit that any intelligence prod- 
uct is more than a snap-shot of knowledge 
believed to be true at that time and that "fin- 
ished intelligence" is, by virtue of the formal 
coordination and review processes, "truer" 
than the pieces of raw intelligence from 
which it was built. The roots of this conceit 
date back to the period when the Intelli- 
gence Community was seen to possess 
domain expertise found neither in its user 
communities nor outside government, and 
its assessments could be considered 
authoritative. During that period, the Sher- 
man Kent posture of standing apart from 
policy users could be considered an appro- 
priate style. Fully coordinated "finished intel- 
ligence" products, such as NIEs, do convey 
authoritativeness as the current, agreed 
judgment of the Intelligence Community. 
These products also carry a sense that they 
and their conclusions can stand the test of 
time. These latter assumptions, however, 
are not necessarily warranted. As most "fin- 
ished" intelligence products involve sub- 
stantial interpretation, analysis, synthesis, 
and judgment, and they are more likely to 
have lengthy inference chains that com- 
pound uncertainty issues in more fragmen- 
tary reporting. 

The validity of the earlier judgments 
expressed in finished products is especially 
important because of the common practice 
of "layering," that is, using previous, formally 
coordinated products as the starting point 
for new assessments and estimates. 16 
Although this practice helps to assure the 
consistency of an analytic line, building on 
established judgments or on prior positions 
of an analyst, branch, group, office, or 
agency is also a reason to be wary. As a 
former senior intelligence official noted, 



15 Sabel, 52ff. 

16 See Conclusion 4, SSCI Report, 22-23. 
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The community 
too often treats 
probing ques- 
tions as attempts 
to "shape" (that 
is, "politicize") 
analyses rather 
than genuine in- 
quiries into the 
quality of evi- 
dence and the 
strength of infer- 
ences. 



Judgments which any analytical actor 
has the greatest incentive to defend 
must be subjected to the most critical 
scrutiny. 17 

The danger arises because, unlike aca- 
demic practice, there is no sustained or 
sanctioned process of after-the-fact ("ex 
post facto" or "ex post") review. Finished 
products are rarely subjected to a consid- 
ered re-examination, nor do they receive 
explicit testing by other parties trying to rep- 
licate their findings — unless there is an obvi- 
ous major intelligence failure. Moreover, 
errors that are recognized subsequent to 
publication are often not corrected, by 
means either of timely notification to readers 
or of corrections fully incorporated through- 
out the knowledge base. 18 Other knowledge 
intensive enterprises, such as law, medi- 
cine, and science, also depend on cumula- 
tive foundational knowledge; but they do far 
better at maintaining the accuracy and cur- 
rency of these critical intellectual resources 
than does the Intelligence Community. 

Even a cursory reading of the report on Iraqi 
WMD highlights the mutually reinforcing 
dangers of the "finished intelligence" conceit 
and of "layering." Too often, the presumed 
authoritativeness of a formal product leads 
users to accept its judgments as established 
and its underlying evidence as validated. 
These products then become the baseline 
for updated assessments, and, as a result, 
the cumulative impact of errors is amplified 
and becomes pervasive. 19 In fact, the con- 
ceit is even more damaging to the effective- 
ness of the Intelligence Community in an 
era when policymakers are increasingly 
likely to be their own "senior analysts" and 



may not believe that "finished intelligence" 
represents the final word on a topic. Today, 
these officials bring other information and 
their own expertise to the task of arriving at 
a comprehensive judgment, and intelli- 
gence material is only one input into that 
process. 20 Yet the community too often 
treats probing questions as attempts to 
"shape" (that is, "politicize") analyses rather 
than genuine inquiries into the quality of evi- 
dence and the strength of inference chains. 



6. The Neglect of Research 

This emphasis on current intelligence, with 
its consequent time pressures and the 
methods needed to meet production 
demands, has produced a range of distort- 
ing effects that are not fully recognized. 21 
First, it has severely undercut the ability of 
analysts to do in-depth research by requir- 
ing that most analytic effort be devoted to 
short-term taskings. Second, by denying 
most analysts the opportunity to work on 
deep products under the tutelage of a senior 
mentor, this emphasis has damaged a key 
element of the indispensable apprentice- 
ship process, a sine qua non for effectively 
training and developing professionally com- 
petent intelligence practitioners. Third, in 
order to meet daily requirements, the pro- 
duction aspects of the current intelligence 
cycle have been accorded undue priority. 22 
Fourth, the emphasis on current intelligence 
helped to create an incentives and rewards 
system for the analysts biased towards 
short-term reporting rather than deep analy- 
sis. And, fifth, without explicit management 
support, the time pressures on, and implicit 
incentives for, analysts to focus on current 



17 Private communication from Fritz Ermarth, 23 January 2005. 

18 See the "Curveball Report," cited in the SSCI Report, 482 and 492. 

19 See, for example, the SSCI Report, 32-33 and 484. 

20 Jack Davis, "Paul Wolfowitz on Intelligence-Policy Relations," Studies in Intelligence 39, no. 5 (1996). 

21 See Chapter Four for a more complete discussion 

22 This is also a problem for newspapers, and it explains why newspapers, with their daily production cycle, are 
different from magazines, which can afford to pay more attention to sustained investigative journalism. 
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production have made the pursuit of curios- 
ity difficult. In contrast, within the scientific 
community, "investigator-initiated" research 
is a primary contributor to discovery and 
innovation, as well as a powerful factor in 
validating the research of others. The Intel- 
ligence Community has need of just such a 
procedure. 

Over the past decade, the Intelligence Com- 
munity has made several attempts to redi- 
rect attention to long-term research and 
products that reflect such efforts, but, with- 
out a sustained base of interested custom- 
ers in the senior policy community, these 
have not succeeded. 23 Analysts have seen 
the emphasis on the President's Daily Brief 
(PDB) and similar serial products as a clear 
indication of where interest (and therefore 
success) is to be found. 24 Although some 
suggest that current intelligence can pro- 
vide the basis for deep understanding, it 
seems obvious that attempting to achieve 
this by compiling current reporting is not a 
satisfactory method for producing inte- 
grated, synoptic analyses that are set in full 
context. A better route to meeting the com- 
peting demands of producing both current 
intelligence and deep analytical products 
would be to exploit the expertise and 
domain understanding of the experienced 
analysts doing in-depth research to identify, 
select, extract, and put into context the 
important tidbits from the reporting 
stream — and to use those as a basis for 
more in-depth and sustained exchanges 
with users of intelligence. 



7. The Neglect of Anticipatory 
Intelligence 

The "information revolution" has unsettled 
intelligence officials by providing policymak- 
ers with alternative sources for both cover- 
age of developments and in-depth research 
and understanding on issues of interest to 
them, as well as allowing them to become 
their own intelligence analysts, if they so 
choose. These trends have left Intelligence 
Community managers struggling to emulate 
the success of television as the provider of 
real-time news and has prompted the worri- 
some notion that customers might now see 
intelligence as less relevant. But the com- 
munity must go beyond the current interest 
areas of its customers if it is to perform its 
primary national function of preventing sur- 
prise. In the new and very dynamic geostra- 
tegic environment, perhaps the most 
important element of warning is the anticipa- 
tory function against the unexpected — dis- 
covering new activities that might prove 
inimical to US national interests and obtain- 
ing information about them. Leads to issues 
of this kind are not likely to come from cus- 
tomer requests; as noted, in science, "inves- 
tigator-initiated" research is a particularly 
important driver for discovery and innova- 
tion. 

Correcting this problem demands that the 
leadership of the Intelligence Community be 
prepared to spend real resources on issues 
that are not immediately "customer-rele- 
vant," even when confronted by what seem 
to be limitless demands to focus on high-pri- 
ority subjects. Waiting on the policy commu- 
nity to furnish requirements for anticipatory 
intelligence all but assures that there will be 
serious warning failures — and the commu- 



Waiting on the 
policy communi- 
ty to furnish re- 
quirements for 
anticipatory intel- 
ligence all but as- 
sures that there 
will be serious 
warning failures. 



23 See Douglas MacEachin, "The Tradecraft of Analysis: Challenge and Change in the CIA," Studies in Intelligence 
46, no. 3 (2002): 23-28. It is also interesting to note that the NIE on Iraqi WMD was requested by the Senate Select 
Committee on Intelligence and the Senate Armed Services Committee, not by executive branch policymakers. 

24 See Rob Johnston, Analytic Culture in the U.S. Intelligence Community. Indeed, in response to these concerns, the 
Commission on the Intelligence Capabilities of the U.S. Regarding Weapons of Mass Destruction strongly urged that 
the PDB not become the centerpiece of the DNI's analytic focus. 
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nity will shoulder the blame for the policy- 
makers "surprise." Faced with its concerns 
about diminishing relevance, the anticipa- 
tory role is one the Intelligence Community 
should grasp. At the same time, the dichot- 
omy between current situational awareness 
based on a stream of detailed reporting and 
anticipatory or predictive intelligence is 
more apparent against a new enemy who 
relies on "stilettos and stealth" than it was 
against the Soviet Union, which amassed 
large stocks of advanced weapons whose 
visible and patterned development path 
took well over a decade and then allowed us 
to collect hard evidence on them by exercis- 
ing those capabilities extensively. For a 
community which frequently restates the 
mantra that it is "an evidence-based" cul- 
ture, this divergence of reduced collection 
capabilities against lower signature targets 
and the increasing demands for anticipatory 



judgments resting on long inference chains 
will create an uncomfortable problem in pro- 
ducing these assessments. 



8. The Loss of "Keystone Species" and 
"Intellectual Middleware" 

The relatively recent ecological concept of 
"Keystone Species" denotes organisms that 
play a central role within an environment, 
either as a resource or as a control mecha- 
nism. Within the Intelligence Community, 
the most important of these "keystone spe- 
cies" is perhaps the "journeymen" ana- 
lysts. 25 These experienced analysts (say, 
those with seven or more years of analytic 
experience) are the functional equivalent of 
doctoral students, post-doctoral fellows, 
and assistant professors in academia and of 
the house staff (interns and residents) in 
medical education, both of which are also 
guild/craft systems. Journeymen perform 
the bulk of the work of producing products, 
making incremental improvements in pro- 
cess, teaching the apprentices, and dissem- 
inating knowledge and skills as they move 
to new communities. The journeymen carry 
institutional memory, transmit knowledge to 
junior analysts, and inculcate in them such 
vital professional values as intellectual curi- 
osity, humility, and an ethos of continual 
learning. In addition, they form the core of a 
high-trust social network built on longstand- 
ing prior contacts that is essential to the dif- 
fusion of knowledge within the Intelligence 
Community and the national security com- 
munity as a whole. 

Recognizing the journeymen's role as a "key- 
stone species" helps to explain the severe 
disruptions caused by the disproportionate 
drawdown in their numbers triggered by the 
budget cutbacks of the early 1 990s. To a con- 



25 "Journeyman" is not used in a pejorative sense, but, rather, in the traditional guild meaning of the term. In CIA 
parlance, these are now called "fully qualified" analysts. Military homologues are Navy chief petty officers ("chiefs") 
and Marine gunnery sergeants ("gunnies"). 
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siderable extent, this cadre, with its deep pro- 
fessional expertise, was sacrificed because 
of the disinterest of the policy community; 
nonetheless, the senior management of the 
Intelligence Community allowed it to happen. 
Moreover, increasing pressures for "broad- 
ening" rotational assignments and "up-or- 
out" promotion policies make it more difficult 
to retain the remaining cadre of deeply skilled 
experts as working analysts. 

The loss of journeymen also underscores 
the crucial importance for the analytical pro- 
fession of soft cultural factors and people- 
related processes, such as mentoring. Inad- 
vertently, as mentors disappeared, the 
important socialization and professionaliza- 
tion processes that are essential to train, 
guide, and acculturate the current flood of 
apprentice analysts went with them. A fur- 
ther difficulty is that, absent either emphasis 
on, or demand for, research products, the 
departure of experienced analysts and the 
focus on current intelligence and reporting 
makes it difficult to train and mature new- 
comers in creating such deep products. 

As a consequence, the community also lost 
the "intellectual middleware" 26 that was a 
central element of its knowledge base and 
what one former senior intelligence officer 
terms its "ecology of expertise." 27 Intellec- 
tual middleware is the profound comple- 
mentary understanding of both domain and 
process that is gained from long experience. 
Along with the knowledge base built-up over 
years, middleware is a necessary link 
between current intelligence and deep 
understanding of a domain; it provides the 
essential capabilities necessary for either 
considered in-depth judgments or meaning- 



ful quick-response products. As a confer- 
ence participant with extensive analytic 
experience noted, "Dick Kerr did a study. . .of 
all the analysis that was written on Iraq [by 
the CIA].... He said. ..it was very good, there 
was a lot of detail, there was a lot of informa- 
tion, but he came away from all that analysis 
having no real sense of what Iraq, the coun- 
try, was all about." 28 Of course, working on 
research projects contributes to the produc- 
tion of middleware, which, in turn, serves as 
a resource for all other products and advice; 
this is something that working on a succes- 
sion of current and limited studies can never 
do. 

The Intelligence Community once had a 
fairly standard method for introducing ana- 
lysts to the process of research, which was 
to have them draft a new edition of a lengthy 
product known as a National Intelligence 
Survey (NIS). Working on a project such as 
this created in-depth analytic expertise and 
domain knowledge on a country or area. 
Above all, the knowledge gained by produc- 
ing an NIS would enable an analyst to pro- 
vide policymakers with the essential context 
for current intelligence, to render informed 
judgments, and to produce timely answers 
to important questions. Unfortunately, the 
importance of the NIS process was not rec- 
ognized within the community, and the prac- 
tice was abandoned. 



9. Failure to Develop Analytic Tools and 
Methods for Validation 

Most of the tools available to support analy- 
sis provide help for specific analytic tech- 
niques or intelligence disciplines, such as 



26 This term from the computer science community denotes software that performs intermediary functions (such as 
translation and data exchange) between heterogeneous systems, enabling them to function in an integrated, 
interoperable manner. The connotation of middleware is "glue" that enables not just interoperability but, in this 
context, also cross-fertilization. 

27 This concept of an "ecology of expertise," articulated by Fritz Ermarth, is a powerful tool for thinking about the 
Intelligence Community's role. 

28 See Charlottesville Conference Report, 6 and Kerr, et al. 
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imagery intelligence (IMINT) and signals 
intelligence (SIGINT), rather than for all- 
source analysis. Even so, there are formal 
methodologies taught and employed within 
the community for all-source analysis, espe- 
cially for assessing and marshalling evi- 
dence, and there are structured methods 
that support hypothesis assessment. 29 Con- 
sistent with its "craft" culture, the Intelli- 
gence Community has used practical 
experience rather than formal validation 
methods to assess utility and select these 
tools. Rob Johnston has observed, how- 
ever, that there is a large number of tools 
and methods available to the analytic com- 
munity, but few are consistently 
employed — and none has been tested or 
validated to assure its effectiveness and util- 
ity. 30 As one seasoned analyst commented, 
"... imagine a nuclear powered Navy in 
which all the Reactor Plant Operating Man- 
uals are written by the current ship's engi- 
neer and whenever he got relieved they 
were all shredded and the new engineer 
had to write his own." 31 Of course, as has 
often been noted by students of intelligence, 
that an agency or community opinion is 
wrong does not automatically mean that the 
process followed in reaching that opinion 
was flawed. 32 Again, quoting Charles Allen, 

We're not very good at evaluating the 
quality of intelligence analysis inde- 
pendent of the outcome. We're out- 
come oriented, rather than process 
oriented. 

Conversely when the policy succeeds 
and a desirable outcome occurs, we 
feel satisfied with the conduct of intelli- 



gence and generally look no further. 
The cumulative effect of this process is 
that it undermines the very essence of 
intelligence analysis. 33 

Within the diverse domain of intelligence 
analysis, one would expect that analytic 
methods and tools would vary significantly, 
not only depending on the sources of intelli- 
gence available and the subject matter, but 
also on the epistemology of the question at 
hand— whether the problem is, to use Fritz 
Ermarth's typology, a secret, a mystery, or an 
obscurity. "Secret" means the information 
exists but must be acquired — usually by clan- 
destine collection methods — and interpreted. 
A "mystery" is a question for which a set of 
possible outcomes may be known but whose 
particular outcome can be known only after 
the fact. An "obscurity" involves questions 
that are often unrecognized and not seen to 
be relevant until they are posed explicitly — 
which requires curiosity, a quality often in 
short supply. Data to illuminate obscurities 
are also often available — even if found in non- 
traditional sources that require imagination to 
identify, if procedures to filter signal from 
noise can be developed, and if appropriate 
inference chains can be constructed from the 
evidence. For example, Murray Feschbach 
drew accurate inferences about the internal 
strength of the Soviet system from totally "out- 
of-domain," low-level, openly-available statis- 
tical data on morbidity and mortality published 
by the Soviets themselves. 34 

The Intelligence Community, however, as a 
function of its history and culture, has con- 
structed a "hierarchy of privilege" for infor- 
mation that still gives the most weight to 



29 See, for example, Hughes and Schum, "Evidence Marshalling and Argument Construction." 

30 Johnston, 72. 

31 Private communication from John Bodnar. 

32 Put another way, "There are too many targets and too many ways of attacking them for even the best intelligence 
agencies to discover all threats in time to prevent them from happening." Kerr, et al. 

33 Allen, 3. 

34 See, inter alia, Russia's Health and Democratic Crises and Russia's Demographic and Health Meltdown. 
Feschbach is currently a senior scholar at the Woodrow Wilson Center and Research Professor Emeritus at 
Georgetown University. 
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secret intelligence. Similarly, many users of 
intelligence information accord greater cre- 
dence to reports with higher levels of classi- 
fication. And, of course, the Intelligence 
Community often perceives its comparative 
advantage with decisionmakers in secret 
information that only it can access and sup- 
ply. At the same time, unfortunately, the 
more evidence and judgments are restricted 
in dissemination by compartmentation and 
distribution limitations, the more likely it is 
that questionable judgments will pass 
unchallenged. This is an especially serious 
problem for HUMINT, which, by its nature, 
provides only a very narrow perspective on 
events for which direct confirmation may not 
be available. Yet timely, authoritative, and 
credible HUMINT is sometimes — however 
rarely — the only way to obtain information 
that can determine strategic direction. 



10. The Hindrances of the 
Security Mindset 

The current security mindset is an additional 
impediment to cooperation among and 
within intelligence agencies, between intelli- 



gence agencies and policy agencies at all 
levels of government, and between commu- 
nity analysts and outside expertise resident 
in both non-government and foreign 
sources. This mindset is extremely risk 
averse with respect to potential information 
loss, and it fosters procedures that make it 
difficult to pull together and share files of rel- 
evant information, to bring fresh perspec- 
tives to bear, and to exploit the synergies of 
expert collaboration. This current security 
paradigm views problem domains as dis- 
crete and separable and insists that protec- 
tion of information (and, therefore, sources 
and methods) is more important than effec- 
tive exploitation and cross-fertilization. Two 
problem areas, in particular, are the tradi- 
tional barrier between the intelligence and 
law enforcement functions and between for- 
eign and domestic intelligence. Under the 
pressure of countering the terrorist threat, 
however, these distinctions are eroding. 
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The following sections lay out some funda- 
mental guidelines, or pathways, for trans- 
forming analysis, provide a set of principles 
upon which to base a restructured analysis 
paradigm, and suggest six specific areas for 
attention. 1 To be effective, however, neces- 
sary changes need to be implemented within 
the system, not just identified. Even a cur- 
sory look at the fate of previous Intelligence 
Community reform efforts does not lead to 
expectations of great success in such an 
endeavor. For this reason, the path sug- 
gested here offers the opportunity to make 
some early course corrections that can be 
initiated without formal sanction and with 
minimal changes to existing structures and 
authorities. 2 It also offers the possibility of 
winning support from the professional intelli- 
gence cadre, who must be convinced that 
the measures recommended are both effec- 
tive and sensible. If they are not convinced, 
institutional inertia, if not outright resistance, 
will impede change. For example, groups of 
analysts could take ownership of these 
areas proposed for action, thereby legitimat- 
ing the transformation effort as an organic 
program instead of one imposed from 
above. 3 



Developing a New Concept for Analysis 

Several recent assessments contend that 
the Intelligence Community's problems are 
manifestations of fundamental and system- 
atic problems within each agency's internal 
processes, cultures, and organizational 
structures. 4 Moreover, these shortcomings 
do not result from a flawed community archi- 
tecture or from insufficient directive authori- 



ties and budgetary control over the 
community as a whole — although they may 
be aggravated by them. 

There are two fundamental challenges in 
rebuilding analysis, with a substantial 
degree of contradiction and inherent tension 
between them. The first is to avoid being 
confidently wrong (as in the NIE on Iraqi 
WMD) by staying closer to available evi- 
dence. The second is to provide judgments 
on complex and often unfamiliar adversaries 
and on their likely behavior based on frag- 
mentary and frequently ambiguous informa- 
tion (as in the circumstances leading up to 
the 9/1 1 attack). 5 The community has still not 
made the shift to recognizing that the domi- 
nant intelligence problems are not penetrat- 
ing "denied areas," but rather understanding 
"denied minds." This shift requires rethinking 
not only the types of information that ana- 
lysts need, but also the nature of the infor- 
mation gathering necessary to provide that 
information. In terms of the latter, intelli- 
gence practitioners will need to accept that 
the appropriate mechanisms will not be lim- 
ited to remote clandestine collection sys- 
tems or case officers. 

The larger issue of how to allow a classic 
hierarchical organization to meet the chal- 
lenges of a more fluid environment and 
dynamic networked adversaries must be 
faced squarely, but it is also an effort well 
beyond the scope of this study. Recognizing 
that this factor is important, however, the dis- 
cussion below does focus on directly 
addressing the sources of analytic failures, 
including several problems stemming from a 
hierarchical organization model that has 
been both badly applied and misapplied. 



The community 
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1 A comprehensive discussion of remedial measures is beyond the scope of this paper. 

2 Several reviewers, all of whom are experts in management theory and organizational behavior, suggested this idea 
in parallel. 

3 See Michael Mears, "The Intelligence Community Genome: RIA Paper #3." It is an insightful look at the practical 
challenges of organizational transformation in the Intelligence Community. 

4 See Jeffrey Cooper, "Intelligence & Warning: Analytic Pathologies," and Johnston. 

5 See Kerr, et al. for a more extensive discussion by seasoned analysts on improvements needed in analysis. 
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The primary pur- 
pose of analytic 
effort is "sense- 
making" and 
understanding, 
not producing 
reports 



Principles for a New Paradigm 

A new paradigm should start with a set of 
basic principles to guide the rebuilding pro- 
cess, restore professional standards to the 
conduct of analysis, and help redress the 
effects of evolved dysfunctional practices. 
These principles should be mutually agreed 
between the leadership of the Intelligence 
Community and the cadre of professional 
analysts as the basis for an "analytic com- 
pact" that will affect priorities and taskings 
as well as incentives and rewards for perfor- 
mance. Furthermore, even if users do not 
fully agree, they also should understand 
these principles to be the foundation that 
guides the analytic community in its work. In 
the long-term, it is clearly in the interest of 
policymakers to rebuild the community's 
expertise and knowledge so that intelli- 
gence can provide highly contextualized 
and meaningful judgments, whether on cur- 
rent or future issues. 

These principles ought, above all, to convey 
that "analysis" needs to be construed 
broadly and not solely in a narrow, "reduc- 
tionist" context that seeks to "know" by 
decomposing a phenomenon into its con- 
stituent parts and approaching it analytically 
only on the basis of induction from detailed 
evidence. Many complex phenomena may 
be better comprehended by approaches 
that are based more on synthesis — that is, 
understanding the larger picture — by focus- 
ing on the relationships among the parts 
and on the emergent behavior produced by 
such interactions. These principles — which, 
of course, are not meant to be exhaustive — 
will recognize, therefore, that: 



Philosophy and Values 

• Analysts must have a "duty of curiosity," 
and the analytic process must encourage 
and reward a deep and meaningful 
understanding of the phenomena under 
investigation; 

• Analysts must be responsible for defining 
knowledge needs and, therefore, 
collection requirements; to do this 
effectively, they must understand 
collection capabilities and be sensitive to 
their limitations; 

• Analysts must be active participants in 
developing integrated strategies for 
collection and analysis, seeking 
information instead of being merely 
passive recipients; 

• The primary purpose of analytic effort is 
"sensemaking" and understanding, not 
producing reports; the objective of 
analysis is to provide information in a 
meaningful context, not individual 
factoids; 

• The knowledge discovered and the 
expertise created when an analyst 
researches a problem is at least as 
important as "finished intelligence" 
products that may result; 

• Learning is an activity that is valued highly 
by both analysts and the organization; 

• Not all intelligence need be immediately 
"actionable;" informing decisionmakers 
and enhancing the quality of the decision 
process is a critical objective; 

• Intuition and creative thinking, including 
positing hypotheses to be tested, are as 
important to analysis as evidence-based 
inductive approaches. 
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The Overall Approach 

• There are different analytic problems, and 
there are diverse approaches and 
methods for resolving them; in many 
cases, appropriate and intuitively usable 
tools may enable their consistent use and 
enhance the effectiveness of these 
approaches; 

• There should be no assumed "hierarchy of 
privilege" of sources or analytic methods; 
a wider range of methodologies needs to 
be employed routinely and consistently, 
not seen as exceptional "alternative" 
techniques; 

• Analytic tools are intended to support, not 
supplant, rigorous and structured 
cogitation by human analysts; 

• There should be better access to, and 
exploitation of, open-source information 
as an element of all-source analysis; 

• Deductive hypothesis-based methods 
should be employed more often as a 
complement to traditional evidence-based 
inductive approaches; 

• There should be more use of formal tests 
of diagnosticity of evidence, thereby 
improving the ability to confirm or deny 
hypotheses; 

• Collaboration during the analytical 
process should be routine, not 
exceptional, and workloads should be 
balanced accordingly; 



Review and assessment — including peer 
review — must be an integral element of 
the analysis and should not be conducted 
only after-the-fact; 

There should be greater recognition of the 
propaedeutic and heuristic roles of 
writing — as tools of discovery and 
learning — for the analyst; writing is not just 
a method of transmitting information to the 
user; 

Contrarian methods and "Red Teams" 
should be a routine part of the analytic 
process; 6 

As opposed to the incremental approach, 
in which new evidence is assessed 
piecemeal for its effect on the course of 
judgments made, more use should be 
made of a "stock-take" approach, in which 
the entire collection of evidence is 
reviewed holistically; 7 

"Process watchers," charged with 
recognizing cognitive impediments and 
process failures, should become integral 
parts of analytic teams; 8 

In addition to reviewing products at hand, 
analytic managers and senior Intelligence 
Community overseers should subject the 
knowledge base and domain expertise to 
continual appraisal in order to assess 
whether the scaffolding of evidence and 
the inferential reasoning is sufficiently 
strong to bear the weight of the judgments 
being made and the policy decisions that 
may rest upon them. 9 



6 Composition of the teams, however, might better be rotational and ad hoc. See also Kerr, et al., 52: "Indeed, 
although certain gaps were acknowledged, no product or thread within the intelligence provided called into question 
the quality of basic assumptions...." 

7 The "stock-take" originated in the UK atomic weapons program; it differs fundamentally from the more usual 
practice, which holds that new information will appropriately correct earlier judgments. But this approach, based on 
Bayesian logic, depends on the earlier "priors," which may never be carefully reexamined. 

8 Professor Daniel Kahneman, who teaches at Princeton University, also suggested this approach in the slightly 
different military "command and control" (C2) context in an interview with me in October 2000. Johnston discusses 
this approach at length in his chapter on "Integrating Methodologists into Teams of Experts." 

9 Fritz Ermarth made this suggestion in a communication to me on 23 January 2005. 
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Interaction with Users 



Analysts should 
expect users to 
"pull threads" 
and question 
judgments in 
open dialogue. 



• There may not be a "right" answer, and 
there may be limits on "knowability"; 

• Users are owed an honest assessment of 
the quality of evidence, uncertainties in 
judgments, and the "knowability" of the 
answer; transparency in the logic chain 
and application of evidence is essential; 

• Such assessments can be better 
conveyed through conversation and 
dialogue than through static "finished" 
products; 

• Analysts should expect users to "pull 
threads" and question judgments in open 
dialogue. 



Management and Oversight 

• Management's first responsibility should 
be to remove impediments to analysts' 
ability to function effectively; the core 
management function is to ensure the 
process is being followed and is adapting 
as necessary to changing needs; 

• Commitment by analysts rather than 
enforced compliance by managers must 
become the driving force in renewed 
analytical practices; 

• Management must be knowledgeable 
about the practice of analysis and must 
provide appropriately focused incentives 
for those they supervise; 

• Managers must encourage self- 
awareness, questioning of existing 
procedures, and striving for continuous 
improvement; 



The organization should create a "learning 
environment" — not only for domain 
knowledge, but also for process and 
methodological expertise; 

The organization must encourage not only 
"near-miss" analysis and error detection, 
but also the consistent reporting of 
anomalies and errors; 

There should be toleration of first errors, 
but no tolerance for repeating the same 
mistake; being wrong will happen, but 
failing to learn should be subject to 
sanction. 



Cautions and Precautions 

• Self-awareness of cognitive biases and 
institutional prejudices, as well as a more 
self-reflective manner, should be intrinsic 
elements of an analyst's mindset; 

• There is no "revealed truth," and assertion 
by reference to evidence or previous 
"finished" intelligence products is not 
proof; 

• "Truth" is not held solely by the 
Intelligence Community, either here or 
abroad; 

• "Truth" is also not necessarily the first 
priority of users' questions, but answers 
by analysts should be; 

• Ongoing re-examination and revalidation 
of previous judgments is very important; 
unless care is taken to validate and 
maintain the currency of the library of 
finished intelligence, "layering" poses 
significant dangers for the analytic 
process; 
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• Keeping an open mind — framing multiple 
hypotheses and looking at different 
potential interpretations of "the 
evidence" — is essential if locking in 
premature judgments is to be avoided; 

• Regardless of the apparent rigor of the 
process, an analyst must approach 
analytic issues with "mindfulness" of the 
potential pitfalls in evidence and analytic 
methods that are being applied to a 
particular problem; 10 

• Tools and methodologies should be tested 
and evaluated for analytic effectiveness 
and usability before adoption, and 
analysts should have an important role in 
evaluating them; 

• Because "mindfulness" only goes so far in 
detecting one's own biases and cognitive 
failings, a supervisory function that 
watches over the conduct of the process is 
important; 11 

• Differences in culture and governance 
mechanisms are real — not all human 
actions are determined by Western 
notions of rational thought or 
Enlightenment values — and they affect 
societal and individual preferences and 
decision metrics. 

Additionally, if at all possible, the first audi- 
ence for the knowledge gained should be 
the peer "community of practice" rather than 
the policy users. 12 Too much focus on "serv- 
ing the first customer" may have served to 
shortchange the depth of analysis and the 
rigor that would be demanded by knowl- 



edgeable peers. Moreover, analysis must 
pay more attention to how the knowledge is 
conveyed to the users while, at the same 
time, recognizing that the message should 
rest on a "sound story." The analytic process 
must reinstill a sense that the discipline of 
writing serves to discover the story, not only 
to convey it, and appropriate incentives to 
encourage these behaviors must be insti- 
tuted. 



Getting Started: Six Fundamentals 

This set of principles provides the basis for 
a systematic campaign to enhance analytic 
effectiveness in the three interrelated areas 
of people, process, and technology. The 
process dimension defines the characteris- 
tics and qualities of the enhancements 
needed in the people and technology areas, 
and that is the focus of this study. The peo- 
ple dimension must address, in addition to 
recruitment and training, impediments to 
good work and work practices, retention, 
and professional development, as well as 
appropriate organizational and institutional 
incentives to overcome them. The technol- 
ogy dimension must be centered around 
enabling and facilitating both the people and 
process dimensions, not simply on updating 
the technical infrastructure to move more 
data faster. 

The corrective measures proposed here 
address six processes that are indispens- 
able to restoring the Intelligence Commu- 
nity's capability to perform effective 
intelligence analysis. These proposals 
emphasize: 



If at all possible, 
the first audience 
for knowledge 
gained should be 
the peer "com- 
munity of prac- 
tice" rather than 
the policy users. 



10 See Warren Fishbein and Gregory Treverton, Rethinking "Alternatives Analysis" to Address Transnational Threats. 

11 The standard approach to addressing bias and prejudice in judgment and decisionmaking has been through 
training to recognize one's own cognitive errors. During my research on cognitive impediments to C2 decisionmaking 
for the Defense Advanced Research Projects Agency (DARPA), Professor Daniel Kahneman pointed out that this 
approach has not succeeded, despite more than 25 years of trying. Kahneman was awarded the Nobel Prize in 
Economics for his work on imperfections in decisionmaking. (Personal interview, October 2000.) 

12 This issue was highlighted in Rozak, etal. 
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1 . A reconceptualized set of processes and 
procedures (including tools, methods, and 
practices) for analysis; 

2. An integrated process for recruiting, train- 
ing, educating, and professionalizing ana- 
lysts based on a traditional graduate 
education model emphasizing close men- 
toring; 

3. A new, more interactive process for com- 
munication between users and intelligence 
analysts throughout the intelligence cycle; 

4. A fundamentally revised process for 
establishing "proof," validating evidence 
and judgments, and reviewing those judg- 
ments; 

5. A process for capturing the lessons of 
experience and advancing organizational 
learning; and 

6. A process for continual collaboration and 
sharing. 



A Revamped Analytic Process 

Effective intelligence analysis requires the 
coupling of deep expertise with innovative 
approaches and intuition instead of the con- 
straining formalism of "scientism." Although 
adopting methods of alternative analysis 13 
and setting up red teams are a useful start, 
creating a more coherent structure and a 
demanding, self-reflective analytic process 
must also involve more than calls for lateral, 
out-of-the-box, or non-linear thinking on the 
part of individual analysts. Real change 
must alter the very modes of thought that 
dominate the expectations and practices of 
today's users, managers, and creators of 



all-source analysis. Both "sensemaking" 
and curiosity should be basic elements of 
this transformed paradigm. 14 

Models already exist for such a new para- 
digm, as a review of other domains in which 
failure of work processes can have large 
potential adverse consequences demon- 
strates. These domains fall into two distinct 
categories: 1) where the standardized pro- 
cedures may not be sufficient to prevent 
routine but costly failures and 2) where rou- 
tine procedures are clearly insufficient to 
face extraordinary conditions. In all of these 
models, however, the salient features are a 
high degree of self-awareness, emphasis 
on early error detection and correction, and 
the ethos of a "learning organization." 

In the first category are organizations, such 
as those engaged in manufacturing and 
transaction processing, which specialize in 
maximizing the effectiveness of routine 
operations through continuous attention to 
improvements in the process. For these 
organizations, even small variances in out- 
come are a signal that routine procedures 
need to be adjusted, or significantly altered, 
in order to correct errors that cumulatively 
could become worrisome. These organiza- 
tions have adopted the emphasis of the 
Quality Movement on consistent, continual 
self-examination and improvement; per- 
haps the best known of these is Toyota with 
its formalized kaizen (continuous improve- 
ment) system. Other organizations have 
implemented similar techniques; among 
these are the "5 Whys Approach," which 
focuses on recursive questioning to identify 
the root causes of failure rather than its 
superficial symptoms, and the Six Sigma 
Movement, which emphasizes consistency 
and reductions in process variance. 15 



13 See Fishbein and Treverton, 2, for a succinct and useful discussion of "alternatives analysis." 

14 "Sensemaking" in this context means the ability to perceive, analyze, represent, visualize and make sense of one's 
environment and situation in a contextually appropriate manner. See both Fishbein and Treverton, 3, and Cooper, 
"Sensemaking: Focusing on the Last Six Inches." 

15 See the Six Sigma website, http://www.isixsigma.com/. 
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In the second category, such domains 
include the nuclear power industry, civil avi- 
ation, and aircraft carrier operations. These 
organizations recognize that routine opera- 
tions can produce or face conditions that 
unexpectedly turn extraordinary, with seri- 
ous, adverse consequences. The paradigm 
for organizations of this type is the High 
Reliability Organization (HRO) model. 16 
Some commentators would also include 
hospitals, and especially their high-risk spe- 
cialty units, as HROs because of the large 
consequences of error. Built-in practices — 
formal protocols, structured procedures, 
and self-awareness measures such as mor- 
tality and morbidity (M&M) conferences- 
are a sign that hospitals recognize the risks 
of errors. 17 

A common denominator of both categories 
of organizations is that they "...reliably fore- 
stall catastrophic outcomes through 'mind- 
ful' attention to ongoing operations." 18 As 
Fishbein and Treverton note, quoting Karl 
Weick and Kathleen Sutcliffe, "The unifying 
trait of HROs is that they exhibit the quality 
of 'mindfulness,' defined as: 

...the combination of ongoing scrutiny 
of existing expectations, continuous 
refinement and differentiation of 
expectations based on new experi- 
ences, willingness and capability to 
invent new expectations that make 
sense of unprecedented events, . . . and 
identification of new dimensions of 
context that improve foresight and cur- 
rent functioning. 19 



These organizations address the possibility 
of extraordinary events by building proce- 
dures that are designed to do more than 
maximize effectiveness and efficiency in the 
conduct of routine operations. The organi- 
zation is focused on addressing non-routine 
operations, so that the unexpected 
"...doesn't surprise or disable them" and 
"...coping actions seldom make the situa- 
tion worse." 20 These organizations see 
small errors in routine operations, in addi- 
tion to their role as process signals, as indi- 
cations that organizational compliance and 
managerial oversight are slipping— and that 
such slippage could presage worse failures. 

Both categories of organization understand 
that successful day-to-day operations can 
give rise, in effect, to mindlessness, defined 
as inattention to the environment and to 
internal procedures. In these circum- 
stances, people slip into routines, fail to 
notice changes in a larger context, see new 
phenomena in old categories, and use 
incoming information (even if it indicates 
significant variances) to confirm expecta- 
tions. Mindfulness, on the other hand, 
emphasizes continuous updating and 
assessing alternate interpretations and 
implications of incoming information; even 
small signs of failure can suggest serious 
problems in organizational processes and 
compliance with them. In these organiza- 
tions, according to Wieck and Sutcliffe, 
there is a "preoccupation with failure, both 
past and present"; and there is a concomi- 
tant stress on early error detection and cor- 



16 A number of perceptive commentators have pointed to HRO as a useful model, although often for diverse reasons. 
See Sabel, Johnston, and Stephen Marrin in "Preventing Intelligence Failures by Learning from the Past," and 
Fishbein and Treverton. 

17 The continuing high rates of medical errors and the increasing appearance of "iatrogenic" (that is, physician- 
induced) diseases in hospitals suggest, however, that these organizations have a long way to go before they can be 
fully recognized as HROs. 

18 Karl E. Weick and Kathleen M. Sutcliffe, Managing the Unexpected: Assuring High Performance in an Age of 
Uncertainty, 23. 

19 Fishbein and Treverton, 4, quoting Weick and Sutcliffe, 25. 

20 Weick and Sutcliffe, 42. 
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The process 
watcher function 
is intended to 
focus exclu- 
sively on identify- 
ing errors in the 
analytic process, 
not on alternative 
interpretations of 
the evidence or 
different logic 
chains. 



rection at the lowest levels, as well as 
emphasis on error reporting upwards as 
part of the self-assessment "contract." 21 

Each category contains proven features 
that recommend themselves for inclusion in 
a new synthesis for a transformed intelli- 
gence analysis process. 22 From the High 
Reliability Organization, the feature is 
"mindfulness," which is composed of five 
processes (and sub-processes below 
them): anticipating and becoming aware of 
the unexpected; containing the unexpected; 
near-miss analysis; active management; 
and enhancing containment. The most 
important elements in creating an anticipa- 
tory capability are a reluctance to simplify 
interpretations, which increases blind spots 
by filtering and abstracting key details; rec- 
ognizing the effect of categorization on 
expectations by continually reexamining 
categories and event coding; and reassess- 
ing the basic assumptions and keystones of 
one's analysis and analysis process. In 
addition, the stress on "near-miss analysis" 
is designed to recognize imperfections and 
errors before they cause consequential fail- 
ures. 



ing root causes of errors. This involves look- 
ing beyond the obvious first answer to the 
"what went wrong" question to the serially 
deeper causes until the base source of error 
is found. From medicine, there are two ele- 
ments worth including. First is the practice 
of mortality and morbidity conferences, 
which focus on both near-miss and failure 
analysis. The second is the practice of 
"grand rounds" in which house staff, stu- 
dents, and attending physicians brief and 
review especially difficult cases. Both mea- 
sures underscore the importance of open 
and collaborative communication in identify- 
ing and assessing hard problems, including 
both those that were resolved successfully 
and those that failed. Such open communi- 
cation is essential to both investigatory and 
teaching roles. Finally, the introduction of 
a "process watcher," as suggested by 
Kahneman, is intended to bring a clear and 
unbiased, outside expert's eye to analytic 
teams. The process watcher function, unlike 
that of a Red Team, is intended to focus 
exclusively on identifying errors in the ana- 
lytic process, not on alternative interpreta- 
tions of the evidence or different logic 
chains. 



Both [near-miss and line stoppages in 
a just-in-time system] trigger root- 
cause analysis meant to uncover not 
only the proximate cause of the inci- 
dent, but to eliminate, through rede- 
sign of the organization if necessary, 
the background conditions which gen- 
erated the immediate source of the 
danger. 23 

From the Quality Movement, the synthe- 
sis model adopts the "5 Whys Approach," 
employing a recursive questioning process 
that emphasizes the importance of identify- 



In light of the real challenges to conducting 
complex analysis effectively, it is important 
that the Intelligence Community identify and 
evaluate tools and methodologies that can 
help analysts make sense of complex phe- 
nomena rife with ambiguous or incomplete 
evidence — and then actually provide them. 
The revamped paradigm must also include 
processes that are more specifically 
directed toward strengthening the practice 
and content of analytic methods. This para- 
digm would very likely incorporate more 
widespread and routinized use of formal- 
ized techniques, such as Analysis of Com- 



21 Weick and Sutcliffe, quoted in Fishbein and Treverton. 

22 This study will develop a brief outline of such organizational characteristics, but a comprehensive treatment will 
require a separate paper. 

23 Sabel, 30. 
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peting Hypotheses (ACH), to explore 
multiple hypotheses and would employ 
appropriate supporting tools to facilitate 
their use. 24 There would also be more 
emphasis on the use of negative evidence 
and on its implications for key assumptions 
and inferences, especially when the orienta- 
tion is toward current intelligence. 25 Finally, 
new collaborative mechanisms, such as 
"blogs" (web logs), "wikis" (wikipedia 
entries), and groupware could be employed 
to facilitate better communication and a 
more continuous dialogue among the par- 
ties within the community of interest. 26 

This new analytic paradigm is designed to 
lead analysts to reflect more intensively on 
the practice of their trade. It is also intended 
to develop more structured procedures and 
to instill in both individual analysts and ana- 
lytic units the discipline to follow them. Such 
changes will need to be implemented care- 
fully, so as not to interfere with individual 
creative processes and existing analytic 
practices that are effective, especially those 
that are unorthodox and not easy to assess 
with formalized metrics. It will be especially 
important to guard against the danger that 
too much introspection will cause analysts 
to avoid risk by dodging judgments. 



Recruiting, Training, Educating, and 
Developing Professionals 

In the wake of 9/11 , the Intelligence Com- 
munity has been able to take advantage of 
an upwelling of public support to tap a large 
pool of candidate analysts. These candi- 
dates are, on the surface, talented, diverse, 



and well educated; but they will require 
extensive training and professionalization in 
order to become effective and productive 
contributors to the analytic community. Fur- 
thermore, in current circumstances, when 
analysts should more frequently be 
addressing the challenge of "discovery" and 
open-ended, undefined problems than more 
readily defined monitoring tasks, they are 
confronted by time pressures that leave 
them with little latitude for reflection and 
wondering. Moreover, our educational sys- 
tem increasingly produces linear thinkers 
more comfortable "painting within the lines" 
and pointed more toward likely solutions 
than toward broader problem-solving capa- 
bilities. 27 It is unclear if these shortcomings 
can be corrected by better-targeted recruit- 
ment and more effective training, or if it must 
be addressed by redesigning fundamental 
processes and practices; it is likely that both 
tracks will be needed. 

However the question of academic prepara- 
tion is resolved, rebuilding the apprentice- 
ship and mentoring system is crucial. It is 
essential, therefore, to reconstitute the "key- 
stone species" represented by the journey- 
man analyst. 28 This cannot be done by 
bringing in a flood of young, inexperienced 
analysts; dousing them with a short period 
of classroom training (begrudged by their 
managers and mostly focused on the right 
tone and format for reports); and leaving 
them to learn good practices while sifting for 
nuggets of current intelligence. 

There is no known mechanism that can turn 
the apprentices into "instant" journeymen; 
they cannot be transformed by lectures, 



24 An example is the ACH Tool developed by Palo Alto Research Center (PARC) for the Novel Intelligence from 
Massive Data (NIMD) Program under the Intelligence Community's Advanced Research and Development Activity 
(ARDA). 

25 Kerr, et al. 

26 See Calvin Andrus's Galileo Award winning paper, "Toward a Complex Adaptive Intelligence Community," in 
Studies in Intelligence 49, no. 3 (2005). 

27 As Daniel Goleman noted in his answer to the "2005 Edge Question," an annual survey by The Edge, an internet 
site favored by the technical community. 

28 Additional incentives to retain these experts appear worth considering. 
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As an Intelligence Analyst's Rite of Passage Might Look 

As the Stefiks made admirably clear in their book about innovation: 

Graduate school is a rite of passage for becoming researchers 
and inventors. Graduate schools create the next generation of 
researchers and inventors who are primed to step into posi- 
tions in the world of science and innovation. 

The experience of graduate school draws on a much earlier 
tradition than undergraduate education, or even high school 
and grammar school. Education prior to graduate school is 
dominated by a program of lectures, exercises and exams. 
Such educational practices have a predetermined curriculum 
intended to serve classes of students essentially in lock 
step.... 

In contrast, graduate school is based on the older tradition of 
mentoring and apprenticeship. Graduate education is about 
assisting students to take on a professional practice. The cur- 
riculum is more tailored. Students acquire the practice by 
working with multiple mentors, adjusting the emphasis to fit 
their career objectives. Students discover, sometimes by 
osmosis, elements of practice that would seldom be encoun- 
tered in a classroom setting. Graduation requires demon- 
strated mastery at the level of a practitioner in the field. 1 

1 Stefik and Stefik, 85. 

abstract study, or classroom exercises; nor 
can software tools enable them to substitute 
for more experienced analysts. 29 This chal- 
lenging task demands a focused, directed 
effort at deep analysis within a subject area, 
most likely through a tutoring and appren- 
ticeship model. 30 In addition, revised class- 
room instruction, based on a significantly 
strengthened curriculum emphasizing ana- 
lytic methodologies and methods, could 
have an important role within a rebuilt pro- 
gram for professional development. The 
Intelligence Community should move away 
from "training courses" that take analysts 
off-line for weeks to months and reintegrate 



this type of training directly into the "prac- 
tice" of intelligence analysis, as is done in 
clinical education for medicine and law. 
Moreover, it is absolutely essential to create 
a professional "duty of curiosity," which the 
training process would embed in the profes- 
sional ethos and management would 
encourage, even in the face of time pres- 
sures to meet priority taskings. Further, a 
peripatetic career for senior analysts — 
"moving around to move up," often increas- 
ingly farther from the actual practice of anal- 
ysis — is not a useful way to foster deep 
expertise or to create effective role models. 
Reestablishing a new cadre of effective pro- 
fessional intelligence analysts will require 
basic changes in their career patterns, 
potentially requiring that the military practice 
of tracking and protecting vital sub-special- 
ties with their own career ladders be emu- 
lated. 

These factors prompt the thrust to restruc- 
ture the career path of intelligence analysts, 
perhaps along the lines of medical or sci- 
ence graduate education — especially doc- 
toral and post-doctoral — that integrates 
classroom learning, research, and clinical 
practice. In neither case are education and 
training separated; each is seen as an inte- 
gral element in producing a practicing mem- 
ber of the medical or scientific community. In 
this context, then, deeper analytical prod- 
ucts are essential to building the process 
expertise and domain knowledge of the 
analyst and to rebuilding the domain knowl- 
edge base. Together, these capabilities 
enable a skilled analyst to contextualize cur- 
rent intelligence for the decisionmaker. 

Such a restructured program should also 
seek to foster an open seminar atmosphere 
during the training process and impart to 
analysts the practice of critical collaborative 
discussion within a professional network 



29 Frank J. Hughes, Preparing for the Future of Intelligence Analysis. 

30 Hughes, 2-3 
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carried directly into the workplace. More- 
over, in this process, in which socialization 
is seen as an important aspect of absorbing 
the ethics and ethos that should guide a pro- 
fessional, a close relationship with a mentor 
is crucial. The mentorship process also rein- 
forces the HRO's emphasis on a "culture of 
learning," a habit of error reporting devel- 
oped by encouraging openness, tolerance 
of even "stupid questions," and professional 
collaboration as a norm. 

The drastic shortages in the cadre of expe- 
rienced analysts prompt three final thoughts 
on this process. First, the Intelligence Com- 
munity must bring back sufficient mentors 
(even if on contracts that permit double-dip- 
ping), so that it can truly support the appren- 
ticeship model with highly personalized 
mentoring. This will not be an easy task; 
mentors will need to be chosen carefully 
and supervised properly, so that the right 
lessons from experience are passed on — 
not cynical views carried away from previ- 
ous experience with a dysfunctional pro- 
cess. Second, the community also needs to 
give up the conceit that it can develop soft- 
ware and tools that will make the novices 
into journeymen or experts without their 
going through this lengthy process of 
apprenticeship. This is not to say, however, 
that appropriate tools either do not exist or 
cannnot be developed to help them do their 
jobs better and perhaps progress through 
the cycle more quickly. Third, and perhaps 
most important in the interim, the Intelli- 
gence Community should look for alterna- 
tive ways to produce the intelligence 
insights that the journeymen used to provide 
while, at the same time, reestablishing the 
vital interrelated processes that created and 
fostered the development of "intellectual 
middleware." During this effort, managers 
must avoid the temptation to use these 



About Apprenticeships 

There is often a gap between what can be learned in formal 
lessons and what needs to be conveyed in total... . 

When graduate students begin working with their mentors, 
they are embarking on a journey with an experienced guide. 
Apprenticeship amounts to going around the research cycle 
a few times, asking questions, and getting help at the trickier 



1 Stefik and Stefik, 86. 

mentors to supplement the analytic cadre or 
to force the apprentices to rush their "analyt- 
ical" fences. 



User-Community Interactions 

The current processes for interaction 
between the Intelligence Community and its 
consumers, especially senior policymakers, 
do not work well at either end of the "Intelli- 
gence Cycle." The community does a less 
than satisfactory job of communicating its 
judgments to these customers. 31 At the 
same time, few users have any real educa- 
tion or understanding of the Intelligence 
Community's capabilities and limitations. To 
a large degree, both problems can be 
ascribed to too little sustained dialogue, 
interaction, engagement, or mutual under- 
standing; both parties fail to understand that 
they are in a collective process of discovery, 
sensemaking, and judgment. These failings 
have not come about by accident, however, 
but by community preference. They flow 
from what have been deeply rooted 
beliefs — especially strong at CIA — that too 
close an association with policymakers and 
their political concerns runs the danger of 
contaminating "pristine" intelligence analy- 
ses with "policy judgments." 32 As noted ear- 



31 The term "Intelligence Cycle" is itself part of the problem. With its Industrial Age antecedents, it usually conveys the 
notion of a self-contained "batch" process rather than a continuous spiral of interactions. 
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lier, maintaining an appropriate degree of 
objectivity is a difficult problem; but a good 
solution is to be found neither by substitut- 
ing users' judgments for those of intelli- 
gence professionals nor by the 
professionals ignoring users' real interests 
and needs. 

Since the collapse of the Soviet Union, this 
mutual lack of understanding, combined 
with the lack of a sustained strategic policy 
that would provide consistent guidance and 
priorities, has forced the Intelligence Com- 
munity to divine targets and priorities from 
immediate customer requirements rather 
than from a longer time horizon with a more 
strategic, synoptic, and open-minded field 
of view. Moreover, this tendency was rein- 
forced throughout most of the 1 990s by bud- 
get pressures that drove the community to 
focus on issues "relevant" to senior policy- 
makers — and, therefore, defensible in bud- 
get hearings. Unfortunately, this approach 
guarantees myopia, and the Intelligence 
Community winds up bearing the blame for 
"failures" to see threats from outside the pol- 
icymakers' fields-of-vision. 

The current "over-the-transom" process for 
taskings and questions often leaves the 
working analyst without a good understand- 
ing of the real issues at stake or the pur- 
poses to which customers will put the 
answers once delivered. Yet, both are 
important if uncertainties and sources are to 
be addressed in a context that the policy- 
maker can appreciate. When the Intelli- 
gence Community provides its analyses 
and judgments, too much emphasis is given 
to the format in which the information is pre- 
sented, and there is too little real dialogue 
with the users. The layout — too frequently 
formal, precisely-formatted, sterile "finished 



products" — often masks uncertainties and 
points of contention unless the reader is wit- 
ting enough to "pull the threads," an effort 
that can lead to charges of politicization. 

The community should move away from the 
notion that "finished intelligence" conveys 
certainty; highlighting and clarifying dis- 
agreements, especially over fundamental 
assumptions and judgments, would be of 
more value to high-level policymakers. 33 At 
the same time, the false expectations con- 
veyed by the "conceit of finished intelli- 
gence" and the "illusion of omniscience" 
must be changed on both sides. Users must 
learn that there may not be a "right" answer; 
that the "more probable" case in the forecast 
set may not be the situation that will eventu- 
ate; and that, therefore, a range of thought- 
ful (and thought through) contingency 
responses may be necessary. 

Equally, analysts and their managers 
should be prepared to be forthright in admit- 
ting what they don't know and in identifying 
explicitly the uncertainties in judgments they 
provide. This means providing greater 
transparency and traceability as they con- 
struct inference chains based on explicitly 
denoted qualified evidence, other informa- 
tion, assumptions, and hypotheses. 34 Ana- 
lysts must themselves remember and 
remind policymakers of Heisenberg's rule: 
that we are not simply outside observers of 
a process who have no effect on the out- 
come; our perceptions and actions are inte- 
gral elements in a multiplayer game with 
strongly coupled feedback loops and action- 
reaction cycles. 35 Finally, it must be recog- 
nized that the frequent lack of deep under- 
standing of intelligence on the part of its 
primary policy users suggests that a serious 
education effort needs to be undertaken. 



32 See Davis, "Kent-Kendall Debate." 

33 See Kerr, et al. on "Integration with the Policy Community," 52. 

34 This has important and challenging implications for the estimative methods used by a community self-defined as 
"evidence-based." 

35 German mathematician Werner Heisenberg (1901-1976). I am indebted to John Bodnar for highlighting this issue. 
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Members of other communities — the strate- 
gic nuclear community may be the best 
example — deal with arcane subjects that 
must be communicated clearly to national 
decisionmakers who often lack experience 
or expertise in such matters. They have 
addressed this problem by developing for- 
mal procedures for educating their users 
and managing their expectations. 

Unfortunately, if all the policymakers see is 
a continuous flow of "current information" 
while lacking deep knowledge of a topic or 
the time to synthesize and integrate what 
they do learn into a coherent picture, an 
Intelligence Community version of Gre- 
sham's law may well apply — factoids devoid 
of context will drive out thinking. This situa- 
tion has implications not only for manage- 
ment of the technical systems that support 
and encourage collaboration and sharing, 
but also for the important social aspects of 
group behavior and the mechanisms for 
interaction between policy users and the 
analysts. It has significant implications both 
for the roles intelligence analysts will play 
and for the modalities through which they 
produce and communicate their analyses to 
users. 

Communicating complex judgments and 
degrees of confidence in those judgments is 
best done through conversation among the 
parties, which demands different mecha- 
nisms than simple dissemination of "facts." 
If the mechanisms for interaction with the 
users of intelligence are designed only to 
support the provision of individual pieces of 
evidence rather than to engage both parties 
in an extended conversation in which ambi- 
guity and subtlety can be communicated, it 
is unlikely that either party will be satisfied 
with these interactions. Indeed, it is worth 
asking whether the Intelligence Community 
has any unique contributions to make to 
anticipatory judgments compared with what 



36 See SSCI Report, 286 and 300. 



policymakers can provide themselves, and, 
if so, what are they? Posing the question 
this way emphasizes that such anticipatory 
assessments are in the realm of judgment, 
an area that the community, in its quest for 
"rigor," has often tried to avoid. Now, given 
the variety of alternative information 
sources available to the policymaker, is 
even superb "current reporting" enough to 
make the community essential, especially to 
the policy users? 



"Proof," Validation, and Review 

Failures in the pre-war estimates concern- 
ing Iraq's WMD capabilities highlight deep- 
seated problems in the extremely "self-ref- 
erential" — that is, customarily internal, colle- 
gial, and lacking rigor — process for 
reviewing and validating intelligence judg- 
ments. At the same time, there is absolutely 
no excuse for allowing easily correctable 
errors, such as "Key Judgments" that differ 
from the body of the text or references to 
earlier assessments that portray their judg- 
ments inaccurately, to be conveyed to con- 
sumers. 36 The existing process relies 
fundamentally on an analyst-level coordina- 
tion process augmented by a hierarchical 
review process, most often by managers 
who possess less specific knowledge and 
are farther removed from the craft of analy- 
sis. The community needs to create new 
processes that capture the best of the legal 
system's adversarial model of open combat 
and the scientific community's truly horizon- 
tal peer review and independent replication, 
sprinkling in alternative analyses and red 
teams to do so on both process and sub- 
stance. 

Such processes, even if conducted totally 
inside the community, are bound to be dis- 
tressing, as knowledgeable individuals sub- 
ject an analyst's evidence, assumptions, 



The community 
needs to create 
processes that 
capture the best 
of the legal sys- 
tem's adversarial 
model and the 
scientific com- 
munity's horizon- 
tal peer review. 
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To understand 
"what works" and 
"what doesn't 
work," the Intelli- 
gence Commu- 
nity should 
establish an insti- 
tutional "lessons- 
learned" process. 



hypotheses, and logic to increased scrutiny. 
These mechanisms, including workshops 
and roundtables, would be useful ; but if they 
are composed solely of Intelligence Com- 
munity members, or those who defer to 
them, then the problem of a self-referential 
"proof process" will continue. Therefore, 
painful as it might prove to be, the Intelli- 
gence Community must step outside its 
usual circle and exploit a wider range of 
expertise. 

In addition to peer review, the scientific 
community has fostered a range of self-cor- 
recting features, such as tension between 
experimentalists and theoreticians. It also 
relies on a wider range of alternative proof 
models. Such additional mechanisms for 
validation should be examined for incorpo- 
ration in the revamped intelligence analysis 
process. Furthermore, as in medicine, law, 
and science, mechanisms need to be devel- 
oped and implemented to ensure that 
knowledge bases are updated and cor- 
rected as necessary and that users are noti- 
fied of major errors in previous reports, so 
that the cumulative knowledge base is as 
accurate as possible. 37 

Finally, both the Intelligence Community 
and its users must have an accurate calibra- 
tion of whether intelligence is "on top" of 
important issues and domains. The review 
process must go beyond assessments of 
individual products and personnel perfor- 
mance to create processes that assess the 
"state of knowledge" and the community's 
(and its users') awareness of that state. This 
type of self-diagnosis is badly needed, 
especially as it creates an environment for 
self-criticism combined with a license to look 
at the adequacy of information both on spe- 
cific issues and across wide domain areas. 



After-Action Reports and Lessons- 
Learned Processes 

Unlike medicine or law, militaries do not 
usually have the luxury of a continuous 
stream of real opportunities to "practice" 
their craft or profession. To address these 
problems, the US military counts on inten- 
sive individual and unit training, institutional- 
ized after-action reports and lessons- 
learned processes, and the exercise of 
complete operational organizations before 
units participate in actual operations. It also 
instantiates understanding of what works 
best as formal "doctrine" that can be studied 
and inculcated to provide a common frame 
of reference and instinctive procedural 
basis. One can find very similar systems 
created in both academia and medicine to 
educate and train their incoming members. 
In these other professions, there is little dis- 
tinction or separation between education, 
research, and training; rather, there is a 
continuum of learning that depends heavily 
on the "hands-on" transmission of domain 
knowledge and process expertise. 

In order to understand both "what works" 
and "what doesn't work," the Intelligence 
Community should establish an institution- 
alized lessons-learned process. This would 
include not only postmortems on major fail- 
ures but also on successes and near 
misses. 38 The purpose of this process is not 
to assign blame, which is traditionally an 
inspector general function, nor is it to pun- 
ish; rather, it should serve as an aid to indi- 
vidual and organizational learning. Both 
after action reports, developed originally at 
the US Army's National Training Center 
(NTC), and lessons-learned processes, 
used by the Army and the US Joint Forces 
Command (JFCOM), are effective methods 
and would be good starting points for creat- 



37 This problem is significantly aggravated by the lack of a coherent information infrastructure. 

38 This is very much like Klein's "premortems," cited in Fishbein and Treverton, 7. See also Gary Klein, Intuition at 
Work: Why Developing Your Gut Instinct Will Make You Better at What You Do, 88. 
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ing an Intelligence Community effort. A 
good complement to the lessons-learned 
process is the use of wargaming and sce- 
nario methods, developed on an accurate 
historical basis, to force the participants to 
examine the situation, the players, their 
interactions, and the outcomes in a thought- 
ful manner. 

After action reports and lessons-learned 
processes can also furnish objective evi- 
dence of the utility of tools and methodolo- 
gies and their suitability for addressing 
various kinds of problems. Without such a 
measured baseline of effective procedures, 
methods, and tools, it is difficult to create 
consistent processes to select and adopt 
appropriate analytic methodologies or to 
train and exercise personnel. In addition, a 
baseline makes credit assignment and per- 
sonnel efficiency reviews more reliable. As 
another former analyst and thoughtful 
observer of the analytic process has written, 

The identification of causes of past 
failure leads to kernels of wisdom in 
the form of process modifications that 
could make the intelligence product 
more useful. A more effective, more 
accurate intelligence capability may 
still be vulnerable to the cognitive and 
institutional pathologies that cause fail- 
ure, but a self-conscious and rigorous 
program based on the lessons derived 
from the existing literature would 
strengthen the intelligence product. 39 



Several of the recent investigations point to 
the important role in these intelligence fail- 
ures of lapses in sharing information and 



Key Questions in Reviewing Lessons Learned 

Veteran analyst Charles Allen provides a list of questions that a 
lessons-learned process can assist in answering: 

• What set of hypotheses was being considered? Was the set 
comprehensive, or was there bias in the selection of 
hypotheses? What a priori probability was attached to each 
hypothesis? Again, was there bias? 

• Was there a good understanding about the observables that 
were expected to differentiate between the hypotheses? Was 
intelligence collection requested on the basis of these 
differentially diagnostic observables? 

• Were all the available data considered? How were the data 
weighted? What degree of credibility was accorded the sources? 

• Was the possibility of deception considered and accounted for? 

• Was the analytic process logically correct? Was the confidence 
in rendered judgments correctly estimated? If so, and if the 
confidence was low, was additional collection requested? 

• Were the judgments presented in a timely and adequate 
manner? 

• And, of course, was intelligence collection responsive and 
timely? 1 



coordinating efforts among the constituent 
elements of the Intelligence Community. Not 
surprisingly, the suggested remedies often 
involve establishing new directive authori- 
ties to mandate coordination and collabora- 
tion and build or improve the technical 
information infrastructures that could sup- 
port collaborative activities. In fact, solutions 
do not start with either directive authorities 
or new IT systems, although an improved, 
technically sophisticated information infra- 



39 Stephen Marrin, "Preventing Intelligence Failures by Learning from the Past," International Journal of Intelligence 
and Counterintelligence, 2004. A recent conference on lessons learned, sponsored by the Center for the Study of 
Intelligence, discussed this question at some length and provides an excellent point-of-departure for implementation 
projects. See Intelligence Lessons Learned Conference. 
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Sharing of private 
information 
among experts is 
task for which 
technical solu- 
tions exist, but 
the community's 
technical, organi- 
zational, cultural, 
and incentive 
structures fail to 
support it. 



structure would help. Rather, effective col- 
laboration is fundamentally a matter of 
culture and values; what is needed is, first, 
to create appropriate incentive structures 
for sharing and, second, to forge expert 
social networks and effective "distributed 
trust" systems. These are problems of orga- 
nizational culture that demand active lead- 
ership at all levels of management 
throughout the community. 

Moreover, as mentioned previously, the 
success of the Intelligence Community 
depends on the promotion of an entire set of 
effective collaborations: among analysts; 
between analysts and collectors; between 
analysts and operations officers; between 
analysts and the intelligence users; and not 
least, between community analysts and 
information sources outside the intelligence 
or national security enterprise. For example, 
tasks such as target assessment and collec- 
tion planning involve complex collaborative 
activity among analysts and collectors. 
Often there are subtle bits of information 
that appear significant only in the context of 
tidbits from other disciplines or aspects of 
tradecraft which may be little understood — 
or not at all — outside the ranks of its own 
practitioners. Each of these collaborations 
involves a distinct "community of interest" or 
"community of practice" and represents a 
different type of social construct; none is 
fundamentally dependent on building more 
elaborate technical infrastructures as its pri- 
mary need. Fostering effective collabora- 
tions can serve an important bootstrapping 
function as well, because participants in the 
collaboration are likely to become effective 
champions of more collaboration. 

Furthermore, true collaboration within the 
Intelligence Community should address 
more than simply sharing raw intelligence 
among different analytic components or tak- 
ing part in the coordination process for fin- 
ished products. The Intelligence Community 
understands how to do these tasks, even if 



they are not implemented effectively today. 
The real challenge and greatest leverage 
will come from sharing private information, 
initial hypotheses and tacit knowledge, 
among networks of experts in order to 
increase the opportunities for discovery of 
previously unrecognized significance. This 
is a task for which technical solutions are 
now known and feasible, but the commu- 
nity's technical, organizational, cultural, and 
incentive structures fail to support it. A true 
collaborative environment must build effec- 
tive trust systems among community ana- 
lysts and collectors, especially between the 
Dl and the DO. Indeed, such trusted envi- 
ronments should extend beyond the Intelli- 
gence Community to policymakers and, 
eventually, to other sources of expertise out- 
side of the community and even outside the 
United States. 

Another important reason for fostering col- 
laboration is that of reducing information 
costs. Such reductions can result not only 
from the powerful impacts of information 
technologies on organizational forms, but 
also from the role of "social networks" as a 
medium for sharing among trusted mem- 
bers. Such networks, especially those 
among journeymen analysts, substantially 
reduce the transaction costs of creating and 
transmitting knowledge; in particular, they 
are a low-cost information resource for the 
essential tacit knowledge, both domain and 
process, that is so difficult to elicit and 
instantiate in formal knowledge systems. 
Moreover, because of the wide-ranging 
social network that journeymen build over 
their years of service, these networks can 
both help to diffuse account-specific 
insights into other areas and infuse other 
perspectives into their domain, often cutting 
across formal community security compart- 
mentation restrictions. As with many exam- 
ples of military C2, it is these informal 
networks and processes that truly enable 
the system to function. The potential of 
using "groupware," such as Groove and 
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other collaboration platforms, in building 
and strengthening these social networks 
bears careful exploration and experimenta- 
tion. 40 

Finally, the Intelligence Community's secu- 
rity mindset should also be addressed. The 
overall approach to security — mandated 
from the top by the newly created DNI if it is 
to have sufficient weight to become the new 
paradigm — must move away from the exist- 
ing risk-averse model, which really seeks to 
avoid problems. Despite the likelihood of 
significant opportunity costs, the direction 
must be toward a risk management model 
like that increasingly being adopted for infor- 
mation security in other government agen- 
cies and the private sector. One approach 
would take responsibility for personnel and 
IT security out of the hands of dozens of 
individual agencies and make them the 



responsibility of a single community man- 
ager who would also have the power to 
adjudicate equities and make risk manage- 
ment decisions. 41 A single authority under 
the new DNI would issue security clear- 
ances and set security requirements for IT 
that would apply throughout the federal gov- 
ernment (or at least the executive branch), 
among government contractors, and, ide- 
ally, to state and local government agen- 
cies, as required. This system would also 
eliminate the need to have clearances 
passed, making expedient collaboration 
easier. Uniform security standards for IT 
systems could make the electronic sharing 
of information and on-line collaboration 
among analysts substantially easier, 
thereby enabling the formation of collabora- 
tive "communities of interest" within a 
secure information environment. 



40 Groove is a software package that creates a virtual private network (VPN) within an information network that 
controls access to a shared collaborative space and provides a variety of tools to facilitate information sharing and 
collaboration. It is accepted by some components of the Intelligence Community as secure and trustworthy, but it is 
not widely employed at this time. 

41 The IRTPA, creating the Information Sharing Executive Program Manager, does mandate this, at least for counter- 
terrorism information. 
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The "Analytic Pathologies" methodology 
expands the typical portrayal of the five-part 
intelligence cycle (planning and direction, 
collection, processing, analysis and produc- 
tion, and dissemination) to include seven 
elements in the analytic process: 

• question specification and subject 
understanding 

• evidence acquisition and situation 
perception/assessment 

• problem/hypothesis specification 



• decision metrics and procedures 

• analysis, assessment, and answer 
preparation 

• communication with the client 

• feedback and post-hoc assessment. 

More consistent with current research on 
judgment and decisionmaking, this expan- 
sion permits a closer look at the analytic pro- 
cess itself. This approach also makes it 
possible to capture explicitly the important 
calibration functions of interactions with cli- 
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ents in framing questions and in delivering 
answers, and it highlights the importance of 
explicit feedback for self-assessment. In 
this approach it is also possible to identify 
more completely the impediments to analy- 
sis and to specify the location and distorting 
effects of each step in the process. 

Finally, the methodology explicitly lays out 
the key processes and functions (as well as 
impediments) at three organizational levels: 
individual analysts, work groups and agen- 
cies, and the Intelligence Community as a 
whole. This layout enables investigators to 
examine the causes of analytic problems and 
locate their organizational sources; these 
problems can occur not only where they 
directly interfere with the analytic process, 
but also in the functions which support 
agency and community analytical processes. 
With that information, appropriate corrective 
measures can be identified more easily. 

Thus, although the Intelligence Community 
organizational level provides direction, pri- 
orities, and controls budgetary resources — 
which are important support mechanisms 



and enablers of output functions — it does 
not itself directly perform any intelligence 
functions, except analyses conducted by 
the National Intelligence Council (NIC), now 
a part of the office of the Director of National 
Intelligence. 

The effectiveness of the Analytic Pathologies 
methodology as a diagnostic tool suggests 
its potential utility when used as a framework 
for developing corrective measures. The 
diagnostic effort highlights the importance of 
properly identifying the organizational level at 
which impediments impinge upon successful 
analytic practice, as well as the level at which 
corrective actions must be taken to restore 
important functions — and these are often not 
the same. For example, there are a substan- 
tial number of impediments to effective per- 
formance by individual analysts, but the 
appropriate corrective measures need to be 
instituted at the group or organizational level 
instead of the analytical level. Different cor- 
rective measures also have different time 
constants of effectiveness to be accommo- 
dated, both in planning and in measuring 
improvement; otherwise, phasing conflicts 
could easily induce resonant oscillation in 
processes and control mechanisms. 

The explicit differentiation among these 
elements allowed by this methodology can 
enable individual and organizations at all 
levels to develop tools to recognize and use 
pathology self-assessments. It also allows 
the development of appropriate measures of 
effectiveness or indicators of performance 
targeted at relevant outputs rather than 
inputs. Thus, extending the Analytic 
Pathologies framework to use as the basis 
for developing corrective measures offers 
promise that it may be able to identify and, 
therefore, avoid perverse consequences that 
might result from altering structures and 
processes while attempting to make them 
more effective. 
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IC Functions and Outputs at Three Levels 



Level 


Key Functions 


Outputs 


Intelligence Community 


• Direction and 
Leadership 

• Setting Priorities 

• Budgetary Control 


• Policies 

• Oversight 

• Budgetary Allocation 


Work Group/Agency 


• Training and 
Acculturation 

• Work Practices and 
Habits (TTPs) 

• Rewards and Incentives 

• Review Processes 

• Organizational 
Knowledge Base 


• Review 

• Production 

• Dissemination 

• Customer Interfaces 

• Work Processes 

• Technical Infrastructure 

• Organizational Culture 


Individual 


• Analytic Expertise 

• Domain Knowledge 


• Answers 

• Hypotheses 

• Knowledge Products 

• Increased Knowledge 

• On-demand Expertise 
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